!
""#$!
Abstract!
We propose a new, filtering approach for solving a large
number of regularized inverse problems commonly found in
computer vision. Traditionally, such problems are solved by
finding the solution to the system of equations that expresses
the first-order optimality conditions of the problem. @is can
be slow if the system of equations is dense due to the use of
nonlocal regularization, necessitating iterative solvers such
as successive over-relaxation or conjugate gradients. In this
paper, we show that similar solutions can be obtained more
easily via filtering, obviating the need to solve a potentially
dense system of equations using slow iterative methods. Our
filtered solutions are very similar to the true ones, but often
up to 10 times faster to compute.
1. Introduction
! %&'()*(!+),-.(/*!0)(!/012(/01340.!+),-.(/*!52()(!,&(6*!
,-7(413'(!3*!1,!)(4,'()!0!.01(&1!'0)30-.(!83'(&!,-*()'(9!3&+:1!
9010;!%&!4,/+:1()!'3*3,&<!0!4.0**34!3&'()*(!+),-.(/!3*!1201!,=!
(*13/013&8!12(!,+1340.!>,5!?@A<!52()(!12(!8,0.!3*!1,!)(4,'()!
12(!0++0)(&1!/,13,&!-(15((&!0&!3/08(!+03);!B(!+),-.(/*!,=!
3/08(!*:+()C)(*,.:13,&<!9(&,3*3&8<!9(-.:))3&8<!93*+0)31D!0&9!
3..:/3&013,&!(*13/013,&!0)(!(E0/+.(*!,=!3&'()*(!+),-.(/*!3&!
3/083&8!0&9!4,/+:1()!'3*3,&!?$AF?"A;!B(!:-3G:31D!,=!12(*(!
3&'()*(!+),-.(/*!=,)!)(0.C13/(!4,/+:1()!'3*3,&!0++.34013,&*!
+.04(*!*38&3H40&1!3/+,)10&4(!,&!(I43(&1!&:/()340.!*,.'()*!
=,)!*:42!3&'()*(!+),-.(/*;!J)09313,&0..D<!0&!3&'()*(!+),-.(/!
3*!=,)/:.01(9!0*!0!)(8:.0)3K(9!,+13/3K013,&!+),-.(/!0&9!12(!
,+13/3K013,&!+),-.(/!12(&!*,.'(9!-D!H&93&8!12(!*,.:13,&!1,!
31*!H)*1C,)9()!,+13/0.31D!4,&9313,&*<!52342!40&!-(!(E+)(**(9!
0*!0!*D*1(/!,=!.3&(0)!L,)!.3&(0)3K(9M!(G:013,&*;!!
! N(4(&1.D<!(98(C+)(*()'3&8!)(8:.0)3K()*!-0*(9!,&!-3.01()0.!
,)!&,&.,40.!/(0&*!5(38213&8!20'(!=,:&9!:*(!3&!/0&D!'3*3,&!
+),-.(/*!?"AF?OA;!P2()(0*!*:42!&,&.,40.!)(8:.0)3K()*!,=1(&!
+),9:4(!-(11()!*,.:13,&*!120&!.,40.!,&(*<!12(D!8(&()01(!9(&*(!
*D*1(/*!,=!(G:013,&*!1201!3&!+)04134(!40&!,&.D!-(!*,.'(9!'30!
*.,5!&:/()340.!/(12,9*!.3Q(!*:44(**3'(!,'()C)(.0E013,&!0&9!
4,&7:801(!8)093(&1*;!R:42!&:/()340.!/(12,9*!0)(!3&2()(&1.D!
31()013'(<!0&9!0)(!*(&*313'(!1,!12(!4,&9313,&3&8!,=!12(!,'()0..!
+),-.(/;!%1()013'(!/(12,9*!*:42!0*!4,&7:801(!8)093(&1*!0.*,!
)(G:3)(!12(!+),-.(/!1,!-(!*D//(1)34!L0&9!*(/3C9(H&31(M;!
! %&!123*!5,)Q<!5(!*,.'(!)(8:.0)3K(9!,+13/3K013,&!+),-.(/*!
,=!12(!=,)/!
!
minimize () =
‖
−
‖
2
2
+ 
∗
!
L@M!
:*3&8!=0*1!&,&C31()013'(!H.1()3&8<!,-'3013&8!12(!&((9!1,!*,.'(!
9(&*(!*D*1(/*!,=!.3&(0)!(G:013,&*!+),9:4(9!-D!8(,9(*34!0&9!
-3.01()0.!)(8:.0)3K()*!=,)!(E0/+.(;!P(!'0.3901(!,:)!0++),042!
,&!12)((!4.0**34!'3*3,&!+),-.(/*S!,+1340.!>,5!L0&9!93*+0)31DM!
(*13/013,&<!9(+12!*:+())(*,.:13,&<!0&9!3/08(!9(-.:))3&8!0&9!
9(&,3*3&8<!0..!,=!52342!0)(!(E+)(**3-.(!3&!12(!=,)/!L@M;!T:)!
H.1()(9!*,.:13,&*!1,!*:42!+),-.(/*!0)(!0..!'()D!*3/3.0)!1,!12(!
12(!1):(!,&(*!0*!*((&!3&!U38:)(!@<!-:1!@V×!=0*1()!1,!4,/+:1(!
3&!*,/(!40*(*;!W,/+0)(9!1,!12(!=0*1!-3.01()0.!*,.'()!?"A<!,:)!
=,)/0.3*/!3*!&,1!*+(43H4!1,!12(!-3.01()0.!)(8:.0)3K()<!0&9!40&!
*,.'(!/,)(!09'0&4(9!3&'()*(!+),-.(/*!*:42!0*!12(!93*+0)31D!
0&9!12(!,+1340.!>,5!(*13/013,&!+),-.(/*;!
!
Solving Vision Problems via Filtering
Sean I. Young
!
Aous T. Naman
"
Bernd Girod
!
David Taubman
"#
sean0@stanford.edu aous@unsw.edu.au bgirod@stanford.edu d.taubman@unsw.edu.au
# !
Stanford University
"
University of New South Wales
!"#$%&'()'*+,-".#'%&#$,/%"0&1'".-&%2&'3%+4,&52'".'-"2"+.'673"8/,,7'
%&9$"%&2'$2".#'"6&%/6"-&'2+,-&%2',":&'8+.;$#/6&'#%/1"&.62)'<&'2+,-&'
6=&'2/5&'673&'+>'3%+4,&52'-"/'?,6&%".#'>+%'/'(@×'23&&1A$3)'
B
&36='*C
'
!
!
B"23/%"67
'
!
!
D36"8
/,
'
'
!
!
B&4,$%%".#
'
!
!
B&.+"2".#
'
!
!
'
G%$&'2+,-&%2'
D$%'?,6&%".#'2+,-&%2'
!
!
""#X!
2. Inverse Problems
! T&(!=(01:)(!,=!/0&D!3&'()*(!+),-.(/*!3*!1201!12(D!(312()!!
9,!&,1!20'(!0!:&3G:(!*,.:13,&<!,)!12(!*,.:13,&!3*!:&*10-.(Y31!!
9,(*!&,1!9(+(&9!4,&13&:,:*.D!,&!12(!3&+:1;!P(!)(=()!1,!*:42!
+),-.(/*!0*!3..C+,*(9;!B()(=,)(<!3&'()*(!+),-.(/*!0)(!,=1(&!
)(=,)/:.01(9!=,)!:&3G:(&(**!0&9!*10-3.31D;!B(!)(=,)/:.013,&!
40&!-(!9(/,&*1)01(9!5312!0!*3/+.(!.(0*1C*G:0)(*!+),-.(/!,=!
12(!=,)/!
!
minimize
(
)
=
‖
−
‖
2
2
,
!
L$M!
3&!52342!∈
𝑛×𝑚
<!∈
𝑚
;!Z),-.(/!L$M!09/31*!3&H&31(.D!
/0&D!*,.:13,&*!52(&!≤<!=03.3&8!12(!:&3G:(&(**!1(*1<!*,!
0!)(=,)/:.013,&!,=!L$M!3*!&((9(9!3&!123*!40*(;!
! ['(&!52(&!> <!+),-.(/!L$M!40&!*13..!=03.!12(!*10-3.31D!
1(*1;!W,&*39()!12(!+),-.(/!3&*10&4(!5312!3&+:1!9010!
!
=
1.0 0.0
1.0 0.0
0.9 0.1
, =
1.0
1.0
1.0
+ ,!
LXM!
=,)!(E0/+.(;!T&(!40&!4,&*39()!!0!+()1:)-013,&!,&!12(!(E041!
)3821C20&9!*39(!'(41,)!,=!Y:&.(**!= <!12()(!3*!&,!'(41,)!
!*:42!1201!= ;!P23.(!+),-.(/!L$M!09/31*!12(!L:&3G:(M!
*,.:13,&!
ls
=
†
= (
∗
)
−1
∗
<!12(!*,.:13,&!-(4,/(*!
:&9:.D!3&>:(&4(9!-D!+()1:)-013,& 3=!!.3(*!0.,&8!0!+0)134:.0)!
93)(413,&;!B3*!93)(413,&!3*!
1
<!52()(!
1
!3*!0!'(41,)!0.,&8!
12(!/3&,)!(38(&C0E3*!,=!
∗
;!B3*!40..*!=,)!0!)(=,)/:.013,&!
,=!L$M!*3/3.0).D!1,!12(!40*(!52()(!≤;!
2.1. Regularization
! %&!4,/+:1()!'3*3,&!+),-.(/*<!!,=1(&!)(+)(*(&1*!0!2399(&!
H(.9!,=!'0)30-.(*!L*:42!0*!12(!*4(&(!9(+12M<!(042!(.(/(&1!,=!
!0**,4301(9!5312!0!+0)134:.0)!+3E(.!.,4013,&!3&!12(!3/08(;!%&!
*:42!+),-.(/*<!L$M!3*!,=1(&!)(=,)/:.01(9!-D!)(8:.0)3K013,&S!
!
minimize () =
‖
−
‖
2
2
+ 
∗
,!
L\M!
3&!52342!∈
+
𝑛×𝑛
!3*!0!L8)0+2M!]0+.0430&!/01)3E!+(&0.3K3&8!
12(!420&8(*!-(15((&!09704(&1!'()134(*<!0&9!+0)0/(1()!> 0!!
*+(43H(*!0!1)09(,^!-(15((&!12(!H9(.31D!,=!12(!*,.:13,&!1,!12(!
3&+:1!(, )!0&9!*,.:13,&!*/,,12&(**;!Z),-.(/!L\M!09/31*!0!
:&3G:(!*,.:13,&!52(&!(
∗
) ∩() = {}<!0&9!123*!
4,&9313,&!2,.9*!:&9()!/,*1!43)4:/*10&4(*!*3&4(!!3*!,=1(&!0!
2382C+0**!,+()01,)!4,))(*+,&93&8!1,!12(!]0+.0430&!/01)3E!,=!
*,/(!8)0+2!52()(0*!
∗
3*!0!.,5C+0**!,+()01,)!L=,)!3/08(!
9(-.:))3&8M<!,)!0!&,&C&(8013'(!9308,&0.!/01)3E!L=,)!93*+0)31D!
0&9!,+1340.!>,5!(*13/013,&M;!R3&4(!+),-.(/!L\M!3*!G:09)0134!
3&!<!31*!*,.:13,&!/0D!-(!(E+)(**(9!3&!4.,*(9!=,)/!4,&43*(.D!
0*!
opt
= (
∗
+ )
−1
∗
;!
! _(*+31(!12(!*3/+.3431D<!12(!,-7(413'(!,=!+),-.(/!L\M!20*!0!
*:I43(&1.D!8(&()0.!=,)/<!0&9!*:310-.D!9(H&3&8! (E+)(**(*!
/,*1!3&'()*(!+),-.(/*!3&!'3*3,&!0&9!3/083&8!.3Q(!9(+12!0&9!
,+1340.!>,5!(*13/013,&<!9(+12!*:+()C)(*,.:13,&<!4,.,)3K013,&!
?`A<!3/08(!3&+03&13&8!?aA<!9(C-.:))3&8!0&9!9(C&,3*3&8!?\A;!bD!
*:310-.D!9(H&3&8!<!12(!,-7(413'(!,=!L\M!(E+)(**(*!-,12!.,40.!
?@A<!?$A!0&9!&,&C.,40.!?"AF?@@A!)(8:.0)31D!1()/*;!Z),-.(/!L\M!
3*!0.*,!*:I43(&1.D!8(&()0.!1,!(E+)(**!&,&CG:09)0134!/,9(.*!
-0*(9!,&<!=,)!(E0/+.(<!W20)-,&&3()!0&9!c:-()!.,**(*;!
! T&(!&,10-.(!&,&CG:09)0134!,-7(413'(!3*!12(!1,10.C'0)3013,&!
=:&413,&!,=!N:93&!et al.!?$A!
!
minimize () =
‖
−
‖
2
2
+
‖
()
‖
1
,!
L"M!
3&!52342!() =
|
|
<!0&9!!3*!12(!93^()(&4(!/01)3E<!*,!1201!
=
∗
;!d.12,:82!L"M!0++(0)*!G:31(!93^()(&1!=),/!L\M<!31!
3*!*2,5&!-D!W20/-,..(!0&9!]3,&*!?@$A!1201!L"M!40&!)(093.D!
-(!*,.'(9!:*3&8!12(!.088(9!93^:*3'31D!/(12,9!L,)!31()013'(.D!
)(C5(3821(9!.(0*1C*G:0)(*M<!52342!*,.'(*!3&!12(!12!31()013,&!
12(!.(0*1C*G:0)(*!+),-.(/!
!
minimize
𝑘+1
() =
‖
−
‖
2
2
+ 
∗
𝑘
,!
L`M!
3&!52342!
!
𝑘
=
∗
(abs(
𝑘
))
†
,!
LOM!
0&9!
𝑘
!3*!12(!/3&3/3K()!,=!
𝑘
;!R3&4(!(042!+),-.(/!L`M!3*!3&!!
12(!*0/(!=,)/!0*!L\M<!5(!9,!&,1!&((9!1,!*(+0)01(.D!4,&*39()!
0!=0*1!/(12,9!=,)!*,.'3&8!L"M;!
2.2. Local vs Non-Local
! R,.'3&8!)(8:.0)3K(9!3&'()*(!+),-.(/*!,=!12(!=,)/!L\M!40&!
-(!1)04(9!-04Q!1,!Z23..3+*!?@XA<!J3Q2,&,'!?@\A<!0&9!J5,/(D!
?@"A<!?@`A!3&!12(!,&(C93/(&*3,&0.!40*(<!52342!50*!(E1(&9(9!
1,!12(!15,C93/(&*3,&0.!40*(!-D!c:&1!?@OA;!d!+,+:.0)!42,34(!
,=!!3&!15,!93/(&*3,&*!3*!,&(!-0*(9!,&!12(!H&31(C93^()(&4(!
L=9M!,)!12(!H&31(C(.(/(&1!L=(M!*1(&43.*<!52342!0)(!!
!
fd
=
−1
−1 4 −1
−1
,
fe
=
−1 −2 −1
−2 12 −2
−1 −2 −1
,!
LaM!
)(*+(413'(.D;!B(!.011()!3*!:*(9!-D!c,)&!0&9!R42:&4Q!?@A;!
! e3.-,0!0&9!T*2()!?aA!9(/,&*1)01(!12(!-(&(H1*!,=!:*3&8!0!
&,&C.,40.!]0+.0430&!=,)!3/08(!9(&,3*3&8!0&9!3&+03&13&8;!d*!
12(!0:12,)*!+,3&1(9!,:1<!12(3)!&,&C.,40.!]0+.0430&!3*!31*(.=!0&!
090+1013,&!,=!8)0+2!]0+.0430&*!,=!?@aA;!e3'(&!0&!C*0/+.(!
3/08(!52,*(!!'()134(*!0)(!
𝑛
<!1 ≤≤<!5(!40&!9(H&(!
0!8)0+2!]0+.0430&!,'()!12(!'()134(*!0*!= −<!5312!!
9(&,13&8!12(!5(3821(9!09704(&4D!/01)3E!,=!*,/(!8)0+2!,'()!
12(!'()134(*!
{
𝑛
}
<!0&9!= ()!3*!12(!9(8)((!/01)3E!
,=!123*!8)0+2;!%&!'3*3,&!0++.34013,&*<!12(!5(3821(9!09704(&4D!
-(15((&!
𝑛
!0&9!
𝑚
!3*!:*:0..D!0!=:&413,&!,=!
‖
𝑛
−
𝑚
‖
<!*,!
,&(!40&!9(H&(!!0*!
𝑚𝑛
= (
‖
𝑛
−
𝑚
‖
)!3&!1()/*!,=!*,/(!
&,&C3&4)(0*3&8!=:&413,&!∈
+
→
+
;!
2.3. Bilateral vs Geodesic
! T&(!&,10-.(!8)0+2!]0+.0430&!3*!3&*+3)(9!-D!12(!*:44(**!,=!
12(!-3.01()0.!H.1()!?@#A<!?$VA;!R:++,*(!5(!20'(!0&!C*0/+.(!
3/08(!∈
𝑁
𝐷
< 52,*(!*0/+.(!.,4013,&*!0)(!12(!+,3&1*!
!,=!0!)(410&8:.0)!8)39!3&!12(!C!+.0&(;!B(!-3.01()0.C*+04(!
)(+)(*(&1013,&!?$@A!,=!12(!3/08(!'()134(*!3*!!
!
""#\!
!
𝑛
=
𝑛
𝑋
𝑛
𝑌
𝑛
𝑍
∈⊕
[
0,255
]
,!
L#M!
3&!52342!
𝑋,𝑌 ,𝑍
!0)(!12(!*40.(*!,=!12(!-3.01()0.!*+04(!3&!12(3)!
)(*+(413'(!93/(&*3,&*;!%=!5(!9(H&(!12(!8)0+2!09704(&43(*!
,'()!
𝑛
!0*!
𝑚𝑛
= e
−|𝐩
𝑛
−𝐩
𝑚
|
2
/2
<!12(&!
†
!0&9!−!0)(!
)(*+(413'(.D<!12(!-3.01()0.!H.1()!0&9!12(!-3.01()0..DC5(3821(9!
8)0+2!]0+.0430&!/01)34(*;!T-*()'(!1201!52(&!
𝑍
= ∞<!!3*!
*3/+.D!0!e0:**30&!-.:)!,+()01,)!5312!*40.(*!
𝑋
!0&9!
𝑌
;!
! d&,12()!8)0+2!]0+.0430&!,=1(&!=,:&9!3&!(98(C+)(*()'3&8!
)(8:.0)3K013,&!3*!,&(!-0*(9!,&!12(!8(,9(*34!93*10&4(;!%&!*:42!
0!40*(<!/01)3E!!3*!9(H&(9!0*!
𝑚𝑛
= e
−geod(𝐩
𝑛
,𝐩
𝑚
)
<!52()(!
geod(
𝑛
,
𝑚
)!3*!12(!93*10&4(!,=!12(!*2,)1(*1!+012!=),/!+,3&1!
𝑛
!1,!+,3&1!
𝑚
!,&!12(!15,C93/(&*3,&0.!/0&3=,.9!9(H&(9!-D!
12(!'()134(*!
{
𝑛
}
;!B01!3*<!
geod
(
𝑛
,
𝑚
)
= min
𝑝,
(
𝐯
𝑖
)
1≤𝑖≤𝑝
: 𝐯
𝑖
∼𝐯
𝑖+1
,
𝐩
𝑛
=𝐯
1
, 𝐯
𝑝
=𝐩
𝑚
|
𝑖
−
𝑖+1
|
𝑝−1
𝑖=1
,
!
L@VM!
3&!52342!∼
′
!/(0&*!1201!!0&9!
′
!0)(!09704(&1!+3E(.*!,&!
12(!15,C93/(&*3,&0.!8)39;!
! R3&4(!-3.01()0.!0&9!8(,9(*34!8)0+2!]0+.0430&*!,=1(&!20'(!
9(8)((*!1201!93^()!04),**!'()134(*<!&,)/0.3K013,&!3*!1D+340..D!
0++.3(9!=,)!/,)(!:&3=,)/!)(8:.0)3K013,&;!B(!/,*1!4,//,&!
=,)/!,=!&,)/0.3K013,&!3*!
=
†/2

†/2
<!)(=())(9!1,!0*!12(!
*D//(1)34C&,)/0.3K(9!]0+.0430&<!0&9!
=
†
<!)(=())(9!1,!
0*!12(!)0&9,/C50.Q!&,)/0.3K(9!]0+.0430&!?@aA<!?$$A;!b0)),&!
et al.!?OA!:*(!12(!R3&Q2,)&C&,)/0.3K(9!=,)/!?$XA<!?$\A!,=!12(!
-3.01()0.C5(3821(9!8)0+2!]0+.0430&;!bD!4,&1)0*1<!]0+.0430&*!
-0*(9!,&!*1(&43.*!LaM!0)(!0.)(09D!&,)/0.3K(9!:+!1,!0!4,&*10&1!
*40.3&8!=041,)!L(E4(+1!+,**3-.D!01!12(!3/08(!-,:&90)3(*M;!
3. Related Work
! P2()(0*!12(!*,.:13,&!
opt
= (
∗
+ )
−1
∗
!,=!L\M!
3*!*3/+.(<!31*!&:/()340.!('0.:013,&!40&!-(!(E+(&*3'(;![E4(+1!
3&!0!20&9=:.!,=!*4(&0)3,*<!
opt
!/:*1!-(!('0.:01(9!31()013'(.D!
:*3&8!&:/()340.!/(12,9*!*:42!0*!*:44(**3'(!,'()C)(.0E013,&!
,)!4,&7:801(!8)093(&1*<!-,12!,=!52342!)(G:3)(!:*!1,!('0.:01(!
12(!/0++3&8*!
∗
!0&9! )(+(01(9.D;!B(!.011()!
/0++3&8!40&!-(!+0)134:.0).D!(E+(&*3'(!1,!('0.:01(!3=!!20*!0!
&,&.,40.!L9(&*(M!/01)3E!*1):41:)(;!f)D.,'C*:-*+04(!/(12,9*!
.3Q(!4,&7:801(!8)093(&1*!099313,&0..D!)(G:3)(!12(!*+(41):/!,=!
∗
+ !1,!-(!4.:*1()(9!=,)!=0*1()!4,&'()8(&4(;!!
3.1. Fast Solvers
! U,)!,+1340.!>,5!(*13/013,&<!f)g2(&-h2.!0&9!f,.1:&!?@@A!
4,&*39()!12(!-3.01()0..DC)(8:.0)3K(9!3&*10&4(!,=!L\M<!-:1!5312!
12(!W20)-,&&3()!+(&0.1D!=,)!)(8:.0)3K013,&;!B(D!(**(&130..D!
:*(!12(!=041!1201!= −<!52()(!!3*!12(!:&&,)/0.3K(9!
-3.01()0.!H.1()<!0&9!('0.:01(!12(!/0++3&8!!(I43(&1.D!
3&*39(!4,&7:801(!8)093(&1*!5312!0!=0*1!3/+.(/(&1013,&!,=!12(!
-3.01()0.!H.1();!c,5('()<!1(&!,)!/,)(!31()013,&*!,=!4,&7:801(!
8)093(&1*!0)(!:*:0..D!)(G:3)(9!('(&!52(&!+)(4,&9313,&3&8!3*!
:*(9<!52342!3*!&,1!0*!(I43(&1!0*!0!&,&C31()013'(!0++),042;!
! b0)),&!0&9!Z,,.(!?"A!+),+,*(!12(3)!-3.01()0.!*,.'()!=,)!12(!
*+(43H4!40*(!52()(!!3*!-3.01()0.C5(3821(9<!0&9! 3*!*G:0)(!
0&9!9308,&0.;!U,)/3&8!12(!]0+.0430&!
= −
!3&!1()/*!,=!
12(!-3C*1,420*1343K(9!
<!12(D!=041,)3K(!
= 
∗
<!52()(!!
0&9! 0)(!12(!*.34(!0&9!12(!-.:)!,+()01,)*!)(*+(413'(.D;!B(D!
)(=,)/:.01(!+),-.(/!L\M!3&!1()/*!,=!= !0*!
!
minimize () =
−
2
2
+ 
∗
(−),!
L@@M!
12(!*,.:13,&!
opt
!,=!52342!3*!,-103&(9!:*3&8!+)(C4,&9313,&(9!
4,&7:801(!8)093(&1*;!B(!*,.:13,&!,=!12(!,)383&0.!+),-.(/!L\M!
3*!H&0..D!,-103&(9!0*!
opt
≈
opt
;!
! d.12,:82!12(!-3.01()0.!*,.'()!+),9:4(*!(I43(&1!*,.:13,&*!
3&!+)04134(<!12(!*,.'()!3*!31()013'(<!0&9!9,(*!&,1!8(&()0.3K(!1,!
+),-.(/*!5312!,12()!(98(C+)(*()'3&8!)(8:.0)3K()*;!d.*,<!12(!
*,.:13,&!
opt
!*:^()*!=),/!-.,4Q!0)13=041*<!)(G:3)3&8!=:)12()!
+,*1C+),4(**3&8!-D!0!*(4,&9!(98(C+)(*()'3&8!H.1()<!0*!*101(9!
-D!12(!0:12,)*!12(/*(.'(*;!
3.2. Fast Filtering
! U0*1!*,.'()*!.3Q(!12(!-3.01()0.!*,.'()!:.13/01(.D!9(+(&9!,&!
12(!0-3.31D!1,!+()=,)/!-3.01()0.!H.1()3&8!(I43(&1.D;!i0&D!=0*1!
-3.01()0.!H.1()3&8!/(12,9*!20'(!-((&!+),+,*(9;!B(D!3&4.:9(!
12(!090+13'(!/0&3=,.9!?$"A<!12(!e0:**30&!_C1)((!?$`A<!0&9!
12(!+()/:1,2(9)0.!.01134(!?$@A!H.1()*;!d..!,=!12(/!(E+.,31!12(!
=041!1201!H.1()3&8!5312!0!.0)8(!Q()&(.!40&!0.*,!-(!0423('(9!-D!
L3M!9,5&C*0/+.3&8!12(!3&+:1<!L33M!H.1()3&8!12(!9,5&C*0/+.(9!
*38&0.!:*3&8!0!*/0..()!H.1()!Q()&(.<!H&0..D!L333M!:+C*0/+.3&8!
12(!H.1()(9!*38&0.;!B3*!*()3(*!,=!,+()013,&*!3*!,=1(&!)(=())(9!
1,!0*!12(!*+.01C-.:)C*.34(!+3+(.3&(;!R:42!0!+3+(.3&(!8:0)0&1((*!
0!4,/+:1013,&0.!4,/+.(E31D!4,&*10&1!3&!12(!*3K(!,=!12(!H.1()!
Q()&(.;!bD!4,&1)0*1<!12(!4,/+.(E31D!,=!0!&0j'(!-3.01()0.!H.1()!
3/+.(/(&1013,&!5,:.9!*40.(!.3&(0).D!5312!Q()&(.!*3K(;!
! R3/3.0).D<!(I43(&1!8(,9(*34!)(8:.0)3K013,&!9(+(&9*!:+,&!
(I43(&1!8(,9(*34!H.1()3&8;!U0*1!3/+.(/(&1013,&*!,=!12(!H.1()!
3&4.:9(!12(!8(,9(*34!93*10&4(!1)0&*=,)/!?$OA!0&9!12(!9,/03&!
1)0&*=,)/*!?$aA;!R3&4(!8(,9(*34!H.1()3&8!)(G:3)(*!4,/+:13&8!
12(!*2,)1(*1!+012!-(15((&!('()D!+03)!,=!'()134(*!L@VM<!0!&0j'(!
3/+.(/(&1013,&!,=!12(!H.1()!5,:.9!-(!G:31(!(E+(&*3'(;!d*!0&!
(E0/+.(<!3=!_37Q*1)06*!0.8,)312/!3*!:*(9!1,!H&9!0..!+3E(.53*(!
*2,)1(*1!+012*<!8(,9(*34!H.1()3&8!5,:.9!20'(!0&!(
3
)!4,*1!
3&!12(!&:/-()!!,=!+3E(.*;!
4. Our Filtering Method
! P(! &,5! +)(*(&1! 12(! /03&!)(*:.1! ,=! ,:)! 5,)Q;!P(! 0**:/(!
1201!!3*!R3&Q2,)&C&,)/0.3K(9!0*!3&!?"A<!0.12,:82!123*!3*!&,1!
)(G:3)(9!3&!+)041340.!3/+.(/(&1013,&*;!B(!+),+,*(9!/(12,9!
,)383&01(*!=),/!12(!,-*()'013,&!1201!5312,:1!)(8:.0)3K013,&<!
!
argmin
𝐮
‖
−
‖
2
2
= argmin
𝐮
‖
(
−
†
)
‖
2
2
,
!
L@$M!
*,!5(!40&!4,&*39()!
†
!*,/(!1)0&*=,)/(9!*38&0.!1,!H.1()!-D!
.(0*1C*G:0)(*!:*3&8!12(!5(3821*!L3&'()*(!4,'0)30&4(!/01)3EM!
!
""#"!
∗
!L-:1!12(!+),-.(/!3*!*13..!3..C+,*(9M;!k,1(!12(!*1):41:)0.!
L3&!4,&1)0*1!1,!12(!&:/()34M!+*(:9,C3&'()*( ,=!∈
𝑛×𝑚
!3*!
9(H&(9!0*
!
†
=
∗
(
∗
)
−1
if ≤
(
∗
)
−1
∗
if > ,
!
L@XM!
*,!52()(0*!)(.013,&*23+!L@$M!0.50D*!2,.9*<!31!3*!8(&()0..D!not
12(!40*(!1201!
‖
−
‖
2
2
=
‖
(−
†
)
‖
2
2
!52(&!> ;!
! l*3&8!12(!5(3823&8!=
∗
<!5(!+),+,*(!1,!,-103&!12(!
*,.:13,&!,=!12(!)(8:.0)3K(9!+),-.(/!L\M!&,&C31()013'(.D!0*
1
!
!
filt
=
†

†
!
= (),!
L@\0M!
!
=
†

∗
,!
3&!52342!!3*!12(!8)0+2!09704(&4D!L,)!.,5C+0**!H.1()M!/01)3E!
,=!;!R039!*3/+.D<!
filt
!3*!H.1()3&8!,=!12(!&0j'(!*,.:13,&!
†
!
,=!12(!3..C+,*(9!+),-.(/!L$M!:*3&8!12(!.(0*1C*G:0)(*!5(3821*!
<!&,)/0.3K(9!-D!!1,!+)(*()'(!12(!/(0&!,=!12(!*38&0.;!B3*!
3*!12(!39(0!,=!&,)/0.3K(9!4,&',.:13,&!?$#A!0++.3(9!1,!*,.'3&8!
)(8:.0)3K(9!3&'()*(!+),-.(/*;!
! U,)!*,/(!3&*10&4(*!,=!+),-.(/!LXM<!12(!5(38213&8!-D!!3*!
&(312()!&(4(**0)D!&,)!9(*3)0-.(;!%&!12(!9(-.:))3&8!3&*10&4(!,=!
+),-.(/!L$M!=,)!(E0/+.(<!12(!,)383&0.!3..C+,*(9!+),-.(/!3*!1,!
*,.'(!−= !=,)!;!B(!-.:)!,+()01,)!!3*!*1):41:)0..D!
L-:1!&,1!&:/()340..DM!3&'()13-.(<!0&9!31!5,:.9!20'(!-((&!7:*1!
0*!'0.39!1,!=,)/:.01(!12(!)(8:.0)3K(9!3&'()*(!+),-.(/!0*
2
!
!
minimize () =
‖
−
†
‖
2
2
+ 
∗
,!
L@"M!
3&!52342!40*(!12(!5(38213&8!-D!!93*0++(0)*;!_(+(&93&8!,&!
12(!0++.34013,&<!5(!12()(=,)(!:*(!12(!9(C5(3821(9!'0)30&1!,=!
,:)!H.1()3&8!*1)01(8D!
!
filt
=
†

†
, = (),!
L@\-M!
4=;!12(!,)383&0.!+3E(.53*(!5(3821(9!=,)/:.013,&!L@\0M;!
! k,1(!1201!*3&4(!!3*!0!.,5C+0**!H.1()<!,:)!0++),042!L@\0M!
3*!'0.39!,&.D!3=!(+ )
−1
!20*!0!.,5C+0**!)(*+,&*(;!B3*!3*!
=,)1:&01(.D!12(!40*(!=,)!/,*1!3&'()*(!+),-.(/*!3&!'3*3,&;!B(!
3/08(!9(-.:))3&8!+),-.(/!3*!:&3G:(!3&!1201!(+ )
−1
!20*!
0!2382C+0**!)(*+,&*(<!0&9!3*!3..C0++),E3/01(9!-D!;!%&!*:42!
0!40*(<!,&(!40&!0++.D!12(!9(C5(3821(9!'0)30&1!,=!,:)!/(12,9!
L@\-M!1,!*,.'(!12(!+),-.(/;!B(!*:++.(/(&1!93*4:**(*!123*!3&!
/,)(!9(103.;!P(!/0Q(!&,!*+(43H4!0**:/+13,&*!)(80)93&8!12(!!
*1):41:)0.!)0&Q!,=!∈
𝑛×𝑚
<!523.(!5(!4,&13&:(!1,!0**:/(!
1201!() ∩() = {}!=,)!0!:&3G:(!*,.:13,&;!
4.1. Analysis when 𝐂 = 𝐈
! J,!,-*()'(!
filt
≈
opt
!3&!L@\0M<!.(1!:*!4,&*39()!0!*3/+.()!
3&*1):413'(!3&*10&4(!,=!+),-.(/!L\M<!52()(!= ;!B(&<!,:)!
*,.:13,&!L@\0M!40&!-(!5)311(&!0*!
filt
= <!0&9!12(!1):(!,&(!
0*!
opt
= !5312!= (+ )
−1
;!R3&4(!!3*!*D//(1)34!
0&9!+,*313'(!*(/3C9(H&31(<!5(!40&!5)31(!= 
∗
<!52()(!
!0)(!12(!(38(&'(41,)*!,=!<!12(!4,))(*+,&93&8!(38(&'0.:(*!
,=!52342!0)(!= (
1
,
2
, . . . ,
𝑁
);!P(!0**:/(!
𝑛
!0)(!
,)9()(9!0*!0 =
1
≤
2
≤. . . ≤
𝑁
≤1;!!
! T&(!40&!,-*()'(!1201!12(!H.1()!= (−)
∗
!20*!12(!
*+(41)0.!H.1()!?$$A!=041,)*!
!
1 = 1 −
1
≥1 −
2
≥. . . ≥1 −
𝑁
≥0,
!
L@`M!
0&9!= (+ )
−1
∗
!20*!12(!=041,)*!
!
1 =
1
1 +
1
≥
1
1 +
2
≥. . . ≥
1
1 +
𝑁
≥
1
2
,!
L@OM!
0**:/3&8!= 1!=,)!*3/+.3431D;!B(!(38(&'0.:(*!,=!!9(40D!
1,50)9*!0!523.(!12,*(!,=!<!1,50)9*!1 2
⁄
;!c,5('()<!5(!40&!
(0*3.D!(G:0.3K(!12(!15,!*+(41)0.!H.1()!)(*+,&*(*!-D!0++.D3&8!
12(!/0++3&8!(+ )/2;!%&!0&D!40*(<!12(D!-,12!20'(!0!
:&31!_W!803&!L,)!0!:&31!H.1()!)(*+,&*(!1,!12(!4,&*10&1!'(41,)!
M!0*!40&!-(!*((&!3&!12(!.(=1C20&9!*39(!,=!L@`MFL@OM!L12(!H)*1!
(38(&'(41,)!,=!!3*!
−1/2
M;!!
! R3&4(!12(!15,!*+(41)0.!=041,)*!(−)!0&9!(+ )
−1
!0)(!
8(&()0..D!&,1!12(!*0/(<!,:)!H.1()(9!*,.:13,&*!
filt
= !0)(!
&(4(**0)3.D!0&!0++),E3/013,&!,=!
opt
= ;!c,5('()<!*:42!
0&!0++),E3/013,&!3*!)(0*,&0-.(!*3&4(!,:)!1):(!,-7(413'(!3*!1,!
,-103&!0!8,,9!*,.:13,&!1,!0!'3*3,&!+),-.(/<!&,1!1,!044:)01(.D!
*,.'(!+),-.(/!L\M!per se;!
4.2. Analysis when 𝐂 ≠ 𝐈 but (block-) diagonal
! ](1!:*!)(.01(!,:)!H.1()!*,.:13,&!
filt
!1,!12(!1):(!
opt
!52(&
!3*!&,!.,&8()!12(!39(&131D!-:1!9308,&0.;!P(!5)31(!=
†
=,)!4,&'(&3(&4(;!B(&<!12(!*,.:13,&!,=!+),-.(/!L\M!-(4,/(*!
opt
= (+ )
−1
!0&9!
filt
=
†
!L@\0M;!R:++,*(!
0..!5(3821*!0)(!3&3130..D!= !0*!3&!\;@;!J,!,-*()'(!2,5!12(!
*,.:13,&!
opt
!420&8(*!52(&!0&!0)-31)0)D!5(3821!
𝑛𝑛
!3*!*(1!1,!
1 −
𝑛
<!5(!3&',Q(!12(!R2()/0&Ci,))3*,&!=,)/:.0!1,!5)31(!!
!
(+ )
−1
= (+ −
𝑛
𝑛
∗
)
−1
!
L@aM!
!
= +
𝑛

𝑛
𝑛
∗
∗
1 −
𝑛
𝑛
∗

𝑛
!
= +
𝑛
𝑛
𝑛
∗
,
3&!52342!
!
𝑛
=
𝑛
1 −
𝑛
𝑛𝑛
, 0 ≤
𝑛
≤1,!
L@#M!
0&9!
𝑛
!3*!12(!12!4,.:/&!,=!12(!39(&131D!/01)3E;!
! B(!(G:0.313(*!L@aM!1(..!:*!1201!*(113&8!
𝑛𝑛
= 1 −
𝑛
!099*!
𝑛
𝑛
𝑛
∗
!1,!<!52342!3*!12(!:&3G:(!097:*1/(&1!8:0)0&1((3&8!
1201!(+
𝑛
𝑛
𝑛
∗
)!20*!0!:&31!),5C*:/;!B3*!097:*1/(&1!
3*!*/0..!3=!!20*!0!.0)8(!(^(413'(!H.1()!*40.(!*3&4(!!
𝑛
∗
= 1
3/+.3(*!12(!(.(/(&1*!,=!
𝑛
!0)(!*/0..;!P(!*3/3.0).D!8:0)0&1((!
1201!12(!H.1()!
†
!20*!0!:&31!),5C*:/!-:1!&,)/0.3K3&8!-D!
†
!(E+.3431.D;!d!*3/3.0)!0)8:/(&1!1,!L@aMFL@#M!/0D!-(!83'(&!
=,)!12(!8(&()0.!'(41,)30.!40*(!52()(!!3*!-.,4QC9308,&0.<!0*!
3*!12(!40*(!3&!12(!,+1340.!>,5!(*13/013,&!+),-.(/;!
1
$%&'()%#)*%#+,&-%)(+#.%&%-/012+%3#
𝐅
†
𝐀
#*1&#2,#
.%01)(,2&*('#),#()%.1)(,2#
-1).(+%&#&%%2#(2#%454#)*%#61+,/(3#718&&9:%(;%0#,.#:<=#-%)*,;&4#
2
>8-%.(+100?3#
𝐇
†
#
(&#+,-'8)%;#8&(25#)*%#).82+1)%;#:@$#(2#)*%#5%2%.10#
+1&%3#,.#%A+(%2)0?#B(1#)*%#CCD#(E#
𝐇
#(&#&*(E)9(2B1.(12)#F%454#1#/08.#,'%.1),.G4#
!
""#`!
5. Robust Estimation
! T:)!H.1()3&8!/(12,9!L@\M!40&!-(!),-:*13H(9!-D!420&83&8!
12(!H.1()!!3&!12(!8)0+2!9,/03&;!bD!0:8/(&13&8!12(!'()134(*!
L#M!,=!12(!:&9().D3&8!8)0+2!0*!
!
𝑛
=
𝑛
𝑋
𝑛
𝑌
𝑛
𝑍
𝑛
𝑈
,!
L$VM!
3&!52342!
𝑛
!3*!12(!12!(.(/(&1!,=!12(!+)('3,:*!*,.:13,&<!0&9!
𝑈
!3*!12(!*40.(!,=!123*!*,.:13,&;!LU,)!12(!,+1340.!>,5!0&9!12(!
3..:/3&013,&!(*13/013,&!+),-.(/*<!-,12!4,/+,&(&1*!
1𝑛
!0&9!
2𝑛
!0)(!099(9!1,!
𝑛
!5312!12(3)!)(*+(413'(!*40.(*;M!
! %=!!3*!e0:**30&.D!5(3821(9!0*!
𝑚𝑛
= e
−|𝐩
𝑛
−𝐩
𝑚
|
2
/2
<!12(!
3&1),9:413,&!,=!
𝑛
!4,))(*+,&9*!1,!12(!:*(!,=!12(!P(.*42!.,**!
=,)!,:)!)(8:.0)3K013,&;!c,5('()<!3&!12(!40*(!52()(!!3*!12(!
8(,9(*34!H.1()!5312!
𝑚𝑛
= e
−geod(𝐩
𝑛
,𝐩
𝑚
)
<!3&1),9:43&8!
𝑛
!3*!
93I4:.1!1,!3&1()+)(1!53123&!12(!(*10-.3*2(9!),-:*1!(*13/013,&!
=)0/(5,)Q;!R3&4(!12(!P(.*42!.,**!
!
() =
2
(1 −exp(−
2
2
2
⁄
))!
L$@M!
3*!&,&C4,&'(E!L0&9!&,&C2,/,8(&(,:*M<!12(!*40.(!+0)0/(1()!
𝑈
!+.0D*!0&!3/+,)10&1!),.(!3&!8:0)0&1((3&8!12(!4,&'(E31D!,=!
,:)!+),-.(/;!T-*()'3&8!1201!!3*!4,&'(E!04),**!12(!3&1()'0.!
[−, ]<!5(!*2,:.9!*(1!!*:42!1201!/,*1!,=!12(!3&+:1!1,!!=0..!
3&*39(!123*!3&1()'0.;!B(!3&+:1!/0D!=0..!,:1*39(!,=!12(!4,&'(E!
3&1()'0.!*,/(!,=!12(!13/(!0*!.,&8!0*!12(!c(**30&!,=!12(!,'()C
0..!,-7(413'(!3&!L"M!3*!+,*313'(!*(/39(H&31(;!f)g2(&-h2.!0&9!
f,.1:&!?@@A!,&!12(!,12()!20&9!+),+,*(!0&!(I43(&1!/(12,9!1,!
3&4,)+,)01(!,12()!4,&'(E!),-:*1!.,**(*!!3&!L"M;!
6. Solving Vision Problems
! P(!0++.D!,:)!/(12,9!1,!0!&:/-()!,=!'3*3,&!+),-.(/*<!0..!
,=!52342!40&!-(!5)311(&!3&!12(!=,)/!L\M;!T&(!/0D!0.*,!0++.D!
,:)!/(12,9!1,!,12()!*3/+.()!+),-.(/*!93*4:**(9!3&!?"A<!*:42!
0*!*(/0&134!*(8/(&1013,&!0&9!4,.,)3K013,&<!52342!40&!0..!-(!
4,&'()1(9!3&1,!12(!=,)/!L\M;!
6.1. Depth Super-resolution
! %&!12(!9(+12!*:+()C)(*,.:13,&!+),-.(/!?XVAF?X$A<!12(!8,0.!
3*!1,!:+*0/+.(!0!9(+12!/0+!40+1:)(9!-D!0!9(+12!40/()0!1,!0!
2382()!)(*,.:13,&!,&(!3&!0&!(98(C050)(!/0&&();!U,)!0!83'(&!
.,5C)(*,.:13,&!9(+12!/0+!<!12(!*:+()C)(*,.:13,&!+),-.(/!3*!
(E+)(**(9!-D!
!
minimize () =
‖
−
‖
2
2
+ 
∗
,!
L$$M!
3&!52342!12(!9,5&C*0/+.()!= <!52()(!!)(+)(*(&1*!0!
+)(CH.1()!L0!53&9,5(9!sinc<!3&!044,)90&4(!5312!12(!kDG:3*1!
12(,)(/M<!0&9!!3*!0!*:-C*0/+.();!P(!:*(!L@\-M!1,!,-103&!
!
filt
=
†

∗
∗
, = (
∗
∗
),!
L$XM!
*,!5(!H)*1!:+*0/+.(!!:*3&8!
∗
∗
<!H.1()!12(!)(*:.1!:*3&8!
0&9!&,)/0.3K(;!U38:)(!$!3..:*1)01(*!9(+12!*:+()C)(*,.:13,&;!
6.2. Disparity Estimation
! _3*+0)31D!(*13/013,&!40&!-(!=,)/(9!0*!0!&,&C.3&(0)!.(0*1C
*G:0)(*!+),-.(/!01!H)*1<!0&9!*,.'(9!31()013'(.D!0*!0!*()3(*!,=!
.3&(0)L3K(9M!.(0*1C*G:0)(*!+),-.(/*!:*3&8!12(!e0:**Ck(51,&!
0.8,)312/;!%&!12(!+ 112!31()013,&<!12(!(*13/01(9!93*+0)31D!3*!
83'(&!-D!12(!*,.:13,&!,=!12(!)(8:.0)3K(9!3&'()*(!+),-.(/!
minimize
𝑘+1
() =
‖
𝑥
(−
𝑘
) +
𝑡
𝑘
‖
2
2
𝑑
𝑘
(𝐮)
+ 
∗
!
L$\M!
3&!52342!
𝑘
!3*!12(!/3&3/3K()!,=!
𝑘
<!
𝑥
= (
𝑥
)!0&9!
𝑡
𝑘
!
0)(!12(!C!0&9!C9()3'013'(*!,=!12(!3/08(!+03)<!50)+(9!:*3&8!
12(!93*+0)31D!(*13/01(!
𝑘
;!P(!40&!12(&!5)31(!12(!H)*1!1()/!,=!
12(!,-7(413'(!=:&413,&!,=!L$\M!0*!
!
𝑘
(
)
=
‖
𝑥
(
−
𝑘
)
‖
2
2
,
𝑘
=
𝑘
−
𝑥
†
𝑡
𝑘
,
!
L$"M!
:*3&8!12(!)(.013,&*23+!L@$M!,&!
𝑘
();!P(!,-103&!12(!+ 112!
(*13/01(!,=!12(!93*+0)31D!'30!H.1()3&8!
!
𝑘+1
=
†
(
𝑥
2
𝑘
−
𝑥
∗
𝑡
𝑘
), = (
𝑥
2
).!
L$`M!
6.3. Optical Flow Estimation
! B(!,+1340.!>,5!(*13/013,&!+),-.(/!3*!0!'(41,)!(E1(&*3,&!
,=!93*+0)31D!(*13/013,&!L$\M;!R3&4(!12()(!0)(!&,5!15,!'0.:(*!
1,!(*13/01(!01!(042!+3E(.<!,&(!/0D!5,&9()!3=!,:)!/(12,9!40&!
*13..!-(!0++.3(9;!%&!=041<!,:)!=,)/0.3*/!)(/03&*!12(!*0/(;!J,!
)(40+<!12(!+ 112!>,5!(*13/01(!3*!12(!*,.:13,&!,=!!
!
minimize
𝑘+1
() = 
𝑥
∗

𝑥
+ 
𝑦
∗

𝑦
!
L$OM!
!
+
(
𝑥
,
𝑦
)(−
𝑘
) +
𝑡
𝑘
2
2
𝑑
𝑘
(
𝐮
)
!
B"23/%"67'*C
'
!
'
'
'
'
'
C&>&%&.8&'"5/#&'
H%+$.1'6%$6='1"23/%"67'
I+FA%&2+,$6"+.'1"23/%"67
'
D$%'1"23/%"67'J#&+1&2"8K'
D$%'1"23/%"67'J4",/6&%/,K'
!
!"#$%&'L)'M&'
16×
'2$3&%A%&2+,$6"+.'1"23/%"67'5/32'3%+1$8&1'$2".#'6=&'#&+1&2"8'/.1'6=&'4",/6&%/,'-/%"/.62'+>'+$%'5&6=+1'>+%'6=&'
1088 × 1376'
Art'28&.&)'N&26'-"&F&1'+.,".&'47'0++5".#'".)'C&2$,62'/%&'673"8/,'J5+%&'%&2$,62'/%&'/-/",/4,&'".'6=&'2$33,&5&.6K)!
!
""#O!
?XXA<!52()(!= 
𝑥
∗
,
𝑦
∗
∗
<!0&9!
𝑘
<!
𝑥
<!0&9!
𝑦
!0)(!9(H&(9!
*3/3.0).D!0*!=,)!L$\M;!P(!40&!)(5)31(!12(!.0*1!1()/!,=!L$OM!0*!!
!
𝑘
(
)
=
𝑥
,
𝑦
(
−
𝑘
)
2
2
,
!
L$aM!
3&!52342!
𝑘
=
𝑘
−
𝑥
,
𝑦
†
𝑡
𝑘
;!!
! l&.3Q(!3&!12(!93*+0)31D!(*13/013,&!+),-.(/<!5(!&,5!20'(!
15,!>,5!4,/+,&(&1*!1201!40&&,1!-(!H.1()(9!*(+0)01(.DY12(!
3&'()*(!4,'0)30&4(*!
𝑥
,
𝑦
∗

𝑥
,
𝑦
!&,5!4,:+.(!12(!15,!
>,5!4,/+,&(&1*!
𝑥
∗
!0&9!
𝑦
∗
;!B(!,)383&0.!(G:013,&!L@\0M!3*!
12()(=,)(!8(&()0.3K(9!1,!12(!'(41,)30.!40*(;!B(!&(5!(*13/01(!
,=!>,5!3*!,-103&(9!0*!
!
𝑘+1
=
†

𝑘
−
𝑥
,
𝑦
∗
𝑡
𝑘
,!
L$#M!
3&!52342!!
!
=
,!
=
𝑥
2
𝑥
𝑦
𝑥
𝑦
𝑦
2
,!
LXVM!
0&9!
!
=

(

𝑥
2
)


𝑥
𝑦


𝑥
𝑦


𝑦
2
,
LX@M!
*,!1201!0*!5(..!0*!H.1()3&8!12(!*38&0.!
𝑘
−
𝑥
,
𝑦
∗
𝑡
𝑘
<!5(!
&((9!0.*,!1,!H.1()!
𝑥
2
<!
𝑦
2
!0&9!
𝑥
𝑦
;!
! T-*()'(!1201!12(!/0++3&8! *3/+.D!H.1()*!12(!15,!
4,/+,&(&1*! ,=!!*(+0)01(.D<!52()(0*! 12(! /01)34(*!!0&9!!
40&!-(!+()/:1(9!1,!-(!-.,4QC9308,&0.<!52,*(!12!-.,4Q*!0)(!
12(!2 ×2!/01)34(*!
!
𝑛
=
𝑥𝑛
2
𝑥𝑛
𝑦𝑛
𝑥𝑛
𝑦𝑛
𝑦𝑛
2
,
LX$M!
0&9!
!
𝑛
=
𝑛
∗
𝑛
∗
𝑥
2
𝑥
𝑦
𝑥
𝑦
𝑦
2
,!
LXXM!
)(*+(413'(.D;![**(&130..D<!
𝑛
3*!0!5(3821(9!*:/!,=!12(!2 ×2!
3&'()*(!4,'0)30&4(!/01)34(*!5312!52342!1,!&,)/0.3K(!12(!12!
H.1()(9!'(41,);!U38:)(!X!3..:*1)01(*!,+1340.!>,5!(*13/013,&;!
6.4. Image Deblurring
! %&!12(!4.0**34!3/08(!9(-.:))3&8!+),-.(/<!,:)!,-7(413'(!3*!
1,!)(4,'()!0!9(-.:))(9!3/08(!=),/!*,/(!-.:))D!3/08(!;!P(!
:*(!12(!9(C5(3821(9!'0)30&1!L@\-M!,=!,:)!/(12,9!1,!)(4,'()!
12(!9(-.:))(9!3/08(!0*!!
!
filt
=
†

†
, = (),!
LX\M!
3&!52342!!3*!*,/(!Q&,5&!-.:)!,+()01,);!P2(&!= < LX\M!
*3/+.D!)(9:4(*!1,!(98(C050)(!H.1()3&8;!!
! T&(!40&!(E+)(**!
†
= 
†
∗
<!52()(!!3*!12(!93*4)(1(!
15,C93/(&*3,&0.!U,:)3()!-0*3*<!0&9!!3*!12(3)!4,))(*+,&93&8!
/08&31:9(!)(*+,&*(;!P(!40&!4,/+:1(!
†
!3&!12(!=)(G:(&4D!
9,/03&!-D!/:.13+.D3&8!12(!U,:)3()!4,(I43(&1*!,=!!5312!12(!
3&'()*(!/08&31:9(!)(*+,&*(!
†
<!0&9!1)0&*=,)/3&8!12(!)(*:.1!
-04Q!3&1,!12(!3/08(!9,/03&;!
! U,)!+)041340.!3/+.(/(&1013,&*<!2,5('()<!,&(!&((9*!1,!:*(!
12(!&:/()340.!9(H&313,&!,=!
†
;![E+)(**3&8!12(!-.:)!,+()01,)!
D36"8/,'E+F'&26"5/6"+.
'
'
'
'
'
'
'
'
'
H%+$.1'6%$6='
N/2&,".&'>,+F'
D$%'>,+F'J#&+1&2"8K'
D$%'>,+F'J4",/6&%/,K'
!"#$%&'O)'D36"8/,'E+F'J6+3'%+FK'/.1'6=&'8+%%&23+.1".#'E+F'&%%+%'J4+66+5'%+FK'3%+1$8&1'$2".#'6=&'#&+1&2"8'/.1'6=&'4",/6&%/,'-/%"/.62'+>'+$%'
5&6=+1)'M&'4/2&,".&'E+F'"2'PQ(R'/.1'F&'3&%>+%5'O'F/%3".#'"6&%/6"+.2)'<="6&%'3"S&,2'8+%%&23+.1'6+'25/,,&%'E+F'-&86+%2)'
T5/#&'1&4,$%%".#'
'
'
'
'
'
'
'
'
H%+$.1'6%$6='
U+"27'4,$%%&1'"5/#&'
D$%'1&4,$%%&1'J#&+1&2"8K'
D$%'1&4,$%%&1'J4",/6&%/,K'
!"#$%&'Q)'V%+32'+>'6=&'1&4,$%%&1'"5/#&2'>%+5'6=&'W+1/:'1/6/2&6X'3%+1$8&1'$2".#'6=&'#&+1&2"8'/.1'6=&'4",/6&%/,'-/%"/.62'+>'+$%'5&6=+1'
F=&.'
6=&'26/.1/%1'1&-"/6"+.'+>'6=&'4,$%':&%.&,'"2'L)'U+"2&'-/%"/.8&'"2'
10
−5
)'C&2$,62'/%&'673"8/,'J5+%&'%&2$,62'/%&'/-/",/4,&'".'6=&'2$33,&5&.6K)!
!
!
""#a!
0*!= 
∗
<!52()(!!9(&,1(!12(!93*4)(1(!U,:)3()!'(41,)*!
0&9! 3*!12(!9308,&0.!/01)3E!,=!12(!/08&31:9(!)(*+,&*(<!5(!
9(H&(!
ϵ
†
= 
𝜖
†
∗
<!52()(!
!
(
𝜖
†
)
𝑛
=
𝑛
−1
if
𝑛
> ,
0 otherwise
!
LX"M!
3*!12(!12!9308,&0.!(.(/(&1!,=!
𝜖
†
;![**(&130..D<!12(!&:/()340.!
+*(:9,C3&'()*(!
𝜖
†
!1)(01*!0..!
𝑛
≤!0*!0;!P(!40&! )(80)9!,:)!
*,.:13,&!
†

ϵ
†
!0*!0!&,3*(.(**!P3(&()!9(-.:))3&8!*,.:13,&!
H.1()(9!-D!0&!(98(C050)(!H.1()!!52342!3*!12(&!&,)/0.3K(9;!
! d&,12()!42,34(!,=!3&'()*(!H.1()!3*!
ϵ
g
= 
𝜖
g
∗
<!52()(!
!
(
𝜖
g
)
𝑛
= min(
𝑛
−1
,
−1
)!
LX`M!
0&9!,&(!40&!'()3=D!1201!
ϵ
g
!9(H&(9!'30!LX`M!3*!0!8(&()0.3K(9!
3&'()*(!-:1!&,1!12(!+*(:9,C3&'()*(!,=!;!R3&4(!12)(*2,.93&8!
LX"M!3&1),9:4(*!)3&83&8!0)13=041*!3&!12(!9(C-.:))(9!3/08(<!12(!
)(413H(9!H.1()!=041,)*!LX`M!0)(!+)(=()0-.(!,'()!LX"M;!T-*()'(!
1201!12(!8(&()0.3K(9!3&'()*(!
g
!D3(.9*!12(!)(.013,&!
!
argmin
‖
−
‖
2
2
= argmin
‖
(
−
g
)
‖
2
2
LXOM!
*3/3.0).D!1,!12(!)(.013,&!)(80)93&8!
†
3&!L@$M;!U38:)(!"!+.,1*!
12(!+*(:9,C3&'()*(!0&9!12(!8(&()0.3K(9!3&'()*(!)(*+,&*(*;!
7. Experimental Results
! J,!9(/,&*1)01(! 12(!+),+,*(9! /(12,9<! 5(!3/+.(/(&1! ,:)!
H.1()!1,!*,.'(!0!=(5!+),-.(/*!=),/!12(!+)('3,:*!*(413,&;!B(!
93*+0)31D!(*13/013,&!+),-.(/!L$\M!3*!0!*+(430.!40*(!,=!,+1340.!
>,5!(*13/013,&!L$OM<!*,!5(!4,&*39()!12(!.011()!+),-.(/!,&.D!
3&!123*!*(413,&;!P(!:*(!12(! 9,/03&!1)0&*=,)/*!H.1()!?$aA!0&9!
12(!+()/:1,2(9)0.!.01134(!H.1()!?$@A!3/+.(/(&1013,&*!=,)!12(!
8(,9(*34!H.1()<!0&9!12(!-3.01()0.!H.1()<!)(*+(413'(.D;!N:&&3&8!
13/(*!0)(!,-103&(9!,&!0!*3&8.(!4,)(!,=!0&!%&1(.!$;OecK!W,)(!
3O!+),4(**,)!L3i04!/39C$V@@M;!k,1(<!12(!-3.01()0.!'0)30&1*!,=!
,:)!/(12,9*!0)(!*.,5()!120&!12(3)!8(,9(*34!4,:&1()+0)1*!9:(!
*,.(.D!1,!12(!*+((9!,=!12(!-3.01()0.!H.1()!3/+.(/(&1013,&!?$@A!
:*(9;!c,5('()<!12(!-3.01()0.!'0)30&1*!+()=,)/!*.3821.D!-(11()!
120&!12(!8(,9(*34!,&(*;!
! %&!0..!0++.34013,&*<!5(!=,)/:.01(!12(!8)0+2!'()134(*!3&!12(!
CCCCC!*+04(!0*!
!
𝑛
=
𝑛
𝑋
𝑛
𝑌
𝑛
𝑈
𝑛
𝑍
𝑛
𝑍
𝑛
𝑍
,
LXaM!
0&9!,+13/3K(!
𝑋,𝑌
<!
𝑍
!0&9!
𝑈
!:*3&8!8)39!*(0)42!*(+0)01(.D!
=,)!(042!+),-.(/;!B(!)(*:.1*!=,)!,:)!15,!31()013'(!*,.:13,&*!
3&!J0-.(*!@!0&9!$!L8(,9(*34!0&9!-3.01()0.M!0)(!4,/+:1(9!5312!
4,&7:801(!8)093(&1*!L$\!31()013,&*<!&,)/!1,.()0&4(!,=!10
−6
M;!!
7.1. Depth Super-resolution
! l*3&8!12(!9(+12!/0+!*:+()C)(*,.:13,&!9010*(1!,=!?XVA<!5(!
/(0*:)(!12(!044:)04D!0&9!(I43(&4D!,=!,:)!*:+()C)(*,.:13,&!
/(12,9!-0*(9!,&!,:)!H.1()3&8!=,)/0.3*/;!B(!/(12,9!3*!0.*,!
4,/+0)(9!5312!0!&:/-()!,=!,12()!5(..C+()=,)/3&8!,&(*;!P(!
0**:/(!!3&!L$XM!3*!12(!.0&4K,*X!53&9,5(9!sinc!)(*0/+.3&8!
,+()01,);!P(!*(1!,:)!H.1()!*40.(*!090+13'(.D!:*3&8!
!
𝑋,𝑌
= + 2,
𝑍
= 160
⁄
,
𝑈
= + 10
!
LX#M!
=,)!12(!-3.01()0.!'0)30&1!,=!,:)!/(12,9<!0&9!!
!
𝑋,𝑌
= 3,
𝑍
= 48,
𝑈
= 16
√
!
L\VM!
=,)!12(!8(,9(*34!'0)30&1!L!3*!12(!*:+()C)(*,.:13,&!=041,)M;!
! J0-.(!@!.3*1*!12(!+(0Q!RkN!=,)!12(!9(+12!*:+()C)(*,.:13,&!
G/4,&'()'B&36='2$3&%A%&2+,$6"+.'3&%>+%5/.8&'+>'1"Y&%&.6'5&6=+12!'M&'Z*UC'J1NK'-/,$&2'/%&'+>'6=&'2$33&%%&2+,$6"+.'1"23/%"67'6+'6=&'#%+$.1'
6%$6=)'C$..".#'6"5&2'/%&'>+%'6=&'16×'8/2&)'M&'%&2$,62'>+%'+6=&%'5&6=+12'J?%26'2"S'%+F2K'/%&'4/2&1'+.'6=&'5&/.'29$/%&1'&%%+%2'%&3+%6&1'".'P[R)
H%)*,;&#
I.)#
#
J,,K&#
#
HL/(8&#
#
IB%.1 5%#
D(-%#
2×#
4×#
8×#
16×#
#
2×#
4×#
8×#
16×#
#
2×#
4×#
8×#
16×#
#
2×#
4×#
8×#
16×#
<)*%.#-%)*,;&#
78(;%;#M0)%.#NOPQ#
$(%/%0#12;#R.82#NOSQ#
T*12#et al.NOUQ#
V1.K#et al.#NOWQ
X125 #et al.#NOOQ
C%.&)0#et al.NO!Q#
OY4!O#
OY4"Y#
OY4SW#
OU4UO#
OZ4PU#
OZ4WP#
OP4SS#
OP4"S#
OP4OW#
OP4!!#
OU4"Y#
OP4[U#
OO4!Z#
O"4"O#
O"4UW#
O"4ZU#
OS4S"#
OS4W!#
"[4ZO#
"Z4[S#
"[4UO#
"[4"[#
OW4PP#
OW4PW#
#
SW4US#
S!4ZP#
S!4YO#
S"4OO#
S"4U[#
SS4S[#
O[4S!#
OZ4P[#
O[4"Z#
O[4ZO#
SW4UW#
S!4"S#
OY4SP#
OP4[U#
OU4PZ#
OY4YU#
O[4WW###
SW4"Z#
OP4WO#
OO4[O#
OO4SW#
OS4SW#
OP4PS#
OY4!P#
#
SW4"S#
S!4PU#
S!4YY#
S"4"[#
S"4SU#
SS4YZ#
O[4!O#
OZ4OW#
O[4PS#
SW4"!#
SW4S[#
S!4[Z#
OY4!P#
OP4Y[#
OU4YW#
OZ4WW#
OZ4YW#
O[4[W#
OP4W!#
OO4[P#
OO4UW#
OP4!!#
OP4O"#
OY4"P#
#
O[4![#
O[4[Y#
SW4WP#
O[4[Z#
S!4W"#
S!4ZP#
OY4ZW#
OY4"S#
OY4Z!#
OZ4WP#
OZ4ZY#
O[4"[#
OP4YW#
OS4SZ#
OP4WY#
OP4ZY#
OY4!!#
OY4PU#
O"4[O#
O!4[S#
O"4W!#
O"4P"#
OO4SZ#
OS4OU#
"O4[&#
\#
O4W"&#
"S4!&#
\#
!SW4&#
])%.1)(B%#
J(01)%.10#:,0B%.
3
#NPQ#
7%,;%&(+
4
#F""G#
J(01)%.10
4
#F""G#
SW4!U#
S!4ZW#
SO4W"#
OY4"S#
OY4ZZ#
OZ4P[#
OS4ZY#
OP4S!#
OP4[S#
O!4S!#
O!4UY#
32.26
#
SY4PZ#
SZ4[P#
S[4SO#
SS4YU#
SP4PO#
SP4Y[#
S"4OY#
S"4S!#
S"4[U#
O[4PU#
O[4YS#
O[4YY#
#
SZ4SY#
S[4UY#
49.82
SP4ZW#
46.16
SP4YZ#
43.37
S"4Z!#
SO4"W#
40.84
O[4YZ#
SW4UP#
#
SO4YW#
SP4"P#
SU4""#
SW4Z"#
S!4SP#
S!4[U#
OZ4SS#
OZ4YZ#
O[4"[#
OP4!P#
OP4"Y#
35.82
!4U!&#
!4UW&#
Z4"O&#
<8.&#
7%,;%&(+##
J(01)%.10##
S!4YO#
43.63
OZ4O!#
38.98
OP4Y[#
36.15
O!4UU#
O"4""#
#
S[4WU#
49.72
SP4S[#
45.96
S"4YY#
43.09
O[4YW#
39.87
#
S[4PW#
SZ4Z[#
SU4WY#
SP4YZ#
SO4"O#
S"4[U#
SW4O[#
SW4OW#
#
SP4![#
46.51
S!4YP#
42.26
O[4!U#
39.43
OP4O"#
OP4YU#
0.44s
!4S!&#
!
3
^&(25#𝜎
𝑋,𝑌
= 8, 𝜎
𝐿
= 4, 𝜎
𝑈,𝑉
= 3, 𝜆 = 4
𝑠−1/2
3#1&#&855%&)%;#(2#NPQ4##
4
_%.%3#𝜎
𝑋,𝑌
, 𝜎
𝑍
, 𝜎
𝑈
, 𝜆#1.%#E,82;#/?#&%'1.1)%#5.(;#&%1.+*#E,.#%1+*#&+10%4#
!
𝜖
g
!
𝜖
†
!
𝜖
𝑤
!
!
!"#$%&'[)'\/#."6$1&'%&23+.2&2'+>'/'4,$%':&%.&,'J,&>6K'/.1'1"Y&%&.6'
".-&%2&'%&23+.2&2'J%"#=6K)'M&'<"&.&%'%&23+.2&'ℎ
𝑤
'-/%"&2'25++6=,7''
/8%+22'>%&9$&.8"&2)'M&'32&$1+A".-&%2&'%&23+.2&'ℎ
†
'"2'6=%&2=+,1&1'
6+'0&%+)'D$%'#&.&%/,"0&1'".-&%2&'+.&'ℎ
g
'=/2'/'%&86"?&1'%&23+.2&)'
!
""##!
/(12,9*!0&9!12(3)!):&&3&8!13/(*;!B(!)(*:.1*!3&!12(!1,+!),5*!
L,12()!/(12,9*M!0)(!4,/+:1(9!:*3&8!J0-.(!$!,=!?"A!L=),/!12(!
*:++.(/(&1M;!B(!)(*:.1*!,=!b3.01()0.!R,.'()!?"A!0)(!,-103&(9!
:*3&8!12(!+:-.34.D!0'03.0-.(!4,9(;!T:)!15,!H.1()3&8!/(12,9*!
0)(!@F@VV!13/(*!=0*1()!120&!/,*1!/(12,9*!*+(430.3K(9!1,!12(!
*:+()C)(*,.:13,&!0++.34013,&;!U38:)(!$!*2,5*!,:)!16×!9(+12!
/0+*!,-103&(9!:*3&8!12(!8(,9(*34!'0)30&1!,=!,:)!/(12,9;!
7.2. Optical Flow Estimation
! l*3&8!12(!1)03&3&8!*(1!,=!12(!iZ%CR3&1(.!,+1340.!>,5!9010!
*(1!?X\A<!5(!&,5!4,/+0)(!12(!044:)04D!0&9!12(!(I43(&4D!,=!
,:)!H.1()3&8!/(12,9!5312!12(!31()013'(!'0)3013,&0.!,+13/3K()!
,=![+34U.,5!?X"A!0.*,!:*(9!-D!?X`AF?\VA<!12(!c,)&CR42:&4Q!
?@A!0&9!12(!W.0**34mk]!?\@A!/(12,9*;!T:)!31()013'(!-3.01()0.!
-0*(.3&(!3*!*3/3.0)!1,!?@@A<!-:1!:*(*!12(!P(.*42! .,**! 3&!+.04(!
,=!12(!W20)-,&&3()!.,**!=,)!)(8:.0)31D;!P(!3&3130.3K(!12(!>,5!
L$OM!:*3&8!12(!3&1()+,.013,&!,=!_((+i01423&8!?\$A;!b,12!,:)!
/(12,9!0&9![+34U.,5!:*(!X!,:1()!50)+3&8!31()013,&*;!P(!*(1!
,:)!H.1()!+0)0/(1()*!090+13'(.D!:*3&8!!
!
𝑋,𝑌
= 10,
𝑍
= 12,
𝑈
= 0.5!
L\@M!
=,)!12(!-3.01()0.!'0)30&1!,=!,:)!/(12,9<!3&!52342!!3*!12(!),,1!
/(0&C*G:0)(!/08&31:9(*!,=!12(!3&3130.!>,5!'(41,)*<!0&9!
!
𝑋,𝑌
= 20,
𝑍
= 96,
𝑈
=
!
L\$M!
=,)!12(!8(,9(*34!'0)30&1;!P(!*(1!12(!+0)0/(1()*!,=![+34U.,5!
1,!12(!R3&1(.!*(113&8*;!J0-.(!$!+),'39(*!12(!0'()08(!(&9+,3&1!
()),)*!0&9!12(!):&!13/(*!0=1()!,+1340.!>,5!(*13/013,&;!
! B(!8(,9(*34!'0)30&1!,=!,:)!/(12,9!20*!0!*3/3.0)!0'()08(!
(&9C+,3&1!()),)!0*!12(!'0)3013,&0.!,+13/3K()!,=![+34U.,5!L,)!
*:44(**3'(!,'())(.0E013,&M!523.(!-(3&8!@;a!13/(*!=0*1;!%&!12(!
13/3&8!)(*:.1*<!5(!3&4.:9(!12(!13/(!*+(&1!,&!4,/+:1013,&!,=!
12(!(.(/(&1*!,=! LXVM<!52342!3*!V;@X*!+()!50)+3&8!31()013,&!
=,)!0..!/(12,9*;!U38:)(!X!'3*:0.3K(*!,:)!>,5!(*13/01(*;!
7.3. Deblurring and Denoising
! U,)!9(-.:))3&8<!5(!0**:/(!1201!12(!+,3&1!*+)(09!=:&413,&!
,=!12(!-.:)!3*!Q&,5&;!B(!-.:)!Q()&(.*!5(!:*(!20'(!12(!=,)/!

∗
<!52()(!!3*!0!93*4)(1(!e0:**30&!5312!12(!C1)0&*=,)/!!
!
() = 2
−2𝑛
(1
−1
+ 2
0
+ 1
1
)
𝑛
,!
L\XM!
1201!3*<!12(!bC*+.3&(!Q()&(.!,=!,)9()!2;!d*!!3&4)(0*(*!=),/!
@!1,!a<!5(!3&4)(0*(!
𝑋,𝑌
!0&9!
𝑍
!=),/!\!1,!@V<!0&9!$a!1,!X`!
)(*+(413'(.D!=,)!,:)!8(,9(*34!'0)30&1<!0&9!=),/!X!1,!\<!0&9!#!
1,!@$!)(*+(413'(.D!=,)!,:)!-3.01()0.!,&(;!
! J0-.(!X!+),'39(*!12(!+(0Q!RkN*!,=!12(!9(C-.:))(9!3/08(*!
=,)!93^()(&1!-.:)!Q()&(.*;!U,)!4,/+0)3*,&<!12(!)(*:.1*!=,)!2!
LG:09)0134!)(8:.0)31DM<!Jn!L1,10.!'0)3013,&M!?\XA!0&9!P3(&()C
H.1()(9!*,.:13,&*;!d..!0.8,)312/!+0)0/(1()*!:*(9!3&!93^()(&1!
/,9(.*!0)(!=,:&9!:*3&8!0!8)39!*(0)42;!B(!P3(&()!H.1()!:*(*!
0!:&3=,)/!3/08(!+,5()!*+(41):/!/,9(.;!R(+0)0-3.31D!,=!12(!
-.:)!Q()&(.*!/0D!-(!:*(9!1,!044(.()01(!12(!31()013'(!/(12,9*!
=:)12()!L,:)!13/(*!0)(!=,)!93)(41!$_!9(4,&',.:13,&M;!k,1(!12(!
-3.01()0.!H.1()!3*!&,1!,+13/0.!=,)!9(C&,3*3&8!0*!+,3&1(9!,:1!-D!
b:09(*!et al;!?#A<!52,!9(/,&*1)01(!12(!09'0&108(*!,=!+0142C
-0*(9!H.1()3&8!L&,&.,40.!/(0&*!9(&,3*3&8M!,'()!+3E(.C-0*(9!
H.1()3&8!L-3.01()0.!H.1()M<!*,!5(!40&!0.*,!42,,*(!12(!&,&.,40.!
/(0&*!=,)!;!U38:)(!\!*2,5*!4),+*!,=!,:)!9(-.:))(9!3/08(*;!
8. Conclusion
! %&!123*!+0+()<!5(!*,.'(9!)(8:.0)3K(9!3&'()*(!+),-.(/*!'30!
H.1()3&8;!P23.(!*:42!,+13/3K013,&!+),-.(/*!0)(!1)09313,&0..D!
*,.'(9!-D!H&93&8!0!*,.:13,&!1,!0!*D*1(/!,=!(G:013,&*!52342!
(E+)(**(*!12(!,+13/0.31D!4,&9313,&*<!5(!*2,5(9!1201!12(!041!
,=!*,.'3&8!*:42!(G:013,&*!40&!041:0..D!-(!*((&!0*!0!H.1()3&8!
,+()013,&<!0&9!)(=,)/:.01(9!12(!)(8:.0)3K(9!3&'()*(!+),-.(/!
0*!0!H.1()3&8!10*Q;!P(!+),4((9(9!1,!*,.'(!0!&:/-()!,=!'3*3,&!
+),-.(/*!52342!0)(!1)09313,&0..D!*,.'(9!:*3&8!31()013,&*;!P(!
*2,5(9!1201!12(!+()=,)/0&4(!,=!,:)!/(12,9!3*!4,/+0)0-.(!1,!
12(!/(12,9*!*+(43H40..D!103.,)(9!0&9!3/+.(/(&1(9!=,)!12(*(!
0++.34013,&*;!P(!2,+(!1201!,12()!'3*3,&!)(*(0)42()*!0.*,!H&9!
,:)!0++),042!:*(=:.!=,)!*,.'3&8!12(3)!,5&!'3*3,&!+),-.(/*;!
J08.#
&+10%#
]2'8)#
V:>=#
#
CCD9/1&%;#
#
])%.1)(B%#-%)*,;&#
#
<8.#-%)*,;&#
#
`(%2%.#
𝐿2#
#
D@#
7%,#
J(01)#
#
7%,#
J(01)#
√
0.5#
√
1.0#
√
2.0#
√
4.0#
OW4YW#
"Z4O[#
"U4US#
"P4"Z#
#
OS4ZS#
O!4[O#
"[4SO#
"Y4U"#
OP4SW#
O"4!!#
"[4PO#
"Y4UU#
#
OU4"Y#
O"4[W#
"[4ZU#
"Z4WU#
36.44
O"4Y[#
30.18
28.15
OU4S!#
O"4PU#
"[4ZU#
"Y4ZU#
#
OU4S"#
32.95
OW4!W#
"Z4WY#
OU4"[#
32.95
OW4!O#
"Z4!"#
IB%.1 5%#
"Y4YP#
#
OW4[U#
O!4!Z#
#
O!4YY#
31.89
O!4UY#
31.89
O!4ZY#
D(-%#
\#
W4!W&#
W4!S&#
#
!4SU&#
W4UP&#
S4ZW&#
#
0.07s#
!4UZ&#
#
G/4,&' O)']-&%/#&'3&/:'*UC'J1NK'+>'6=&'1&4,$%%&1'"5/#&2'JW+1/:'
1/6/2&6X'LQ'"5/#&2K)'M&'H/$22"/.'4,$%':&%.&,2'$2&1'/%&'1"28%&6&'NA
23,".&2'+>'+%1&%'2𝑛X'>+%'𝑛 = 1, 2, 4, 8)'M&'.+"2&'-/%"/.8&'"2'10
−5
)'
G/4,&' L)']-&%/#&'^Z^'+.'\ZTA*".6&,'JO'F/%3".#'26/#&2K)'],,'E+F'
"."6"/,"0&1'$2".#'PQ(R)'T.'&/8='F/%3".#'"6&%/6"+.X'^3"8!,+FX'UI'/.1'
_*'$2&'*DC'J"6&%/6"-&KX'F=",&'F&'$2&'.+.A"6&%/6"-&'?,6&%".#)
:%a8%2+%#
]2()(10#
bVb#
])%.1)(B%#&,08)(,2&#
#
<8.&#
_:
5
>c#
b'(+#
7%,
6
#
J(01)
7
#
7%,#
J(01)#
100%?!#
100%?"#
1-/8&*Y#
/1-/,,!#
/1-/,,"#
/12;15%!#
/12;15%"#
+1B%S#
-1.K%)"#
-,82)1(2!#
&*1-12"#
&*1-12O#
&0%%'(25!#
)%-'0%"#
W4Y[Y#
W4YS!#
W4YOZ#
W4Z[O#
!4[U[#
W4[[[#
W4U![#
O4[SW#
!4!WW#
W4Z!Y#
W4P!S#
W4PZ[#
W4SZU#
"4PWZ#
W4SOZ#
W4OZ!#
"4SOU#
W4SYO#
"4O""#
W4[YO#
W4P!U#
P4Z""#
!4!PP#
W4SY!#
W4"O[#
W4"Y[#
W4!OS#
S4POY#
W4"O"#
W4"PY#
W4PYO#
0.335
!4PSO#
0.578
0.294
3.503
0.619
W4SW[#
0.182
W4!ZW#
W4!!W#
1.993
W4"ZW#
0.244
0.538
W4O[W#
!4PU"#
W4U!W#
W4"[U#
O4PUY#
W4UOP#
0.379
W4"WU#
W4!YS#
0.082
"4W!!#
W4"PU#
W4"Y[#
W4P["#
W4OSP#
!4PSO#
W4P[Z#
W4OWS#
O4U!W#
W4UPW#
W4S"[#
W4"!P#
W4![O#
W4!!W#
"4WS!#
W4"SZ#
W4"YO#
W4PUU#
W4OSU#
1.536
W4UWO#
W4OW"#
O4PZY#
W4U"Z#
W4SS"#
W4"WP#
W4!YS#
W4!!!#
"4WO"#
#
0.231
W4"SP#
W4PYY#
W4OSO#
!4PPU#
W4UWW#
W4OWS#
O4PZO#
W4USZ#
W4OZZ#
W4![Z#
W4!Z"#
W4WZY#
"4WOY#
0.228
W4"P"#
W4PS[#
W4OP!#
!4PU!#
W4UWU#
W4OWP#
O4PSS#
W4UOZ#
W4O[W#
W4![!#
0.167
W4W[O#
"4W""#
IB%.1 5%#
!4![S#
!4SS!#
0.772
W4YZS#
W4Y[Y#
W4Y[W#
#
W4YZS#
W4YYZ#
D(-%#
\#
W4PO&#
!Z4"&#
!4![&#
"4U[&#
!S4S&#
#
0.65s
O4O"&#
!
5
^&(25#
𝜆 = 40
#12;#&8++%&&(B%#,B%.9.%01d1)(,2#F:<=G4#
6
^&(25#
𝜎
𝑋,𝑌
= 8, 𝜎
𝑍
= 48, 𝜎
𝑈
= 0.5 + 0.25𝑠
#12;#
𝜆 = 2
4#
7
^&(25#𝜎
𝑋,𝑌
= 6, 𝜎
𝑍
= 10, 𝜎
𝑈
= 0.5 + 0.25𝑠#12;#𝜆 = 24
#
!
"`VV!
References
' P(R' N&%6=+,1' W)' Z)' _+%.' /.1' N%"/.' H)' *8=$.8:)' B&6&%5".".#'
+36"8/,'E+F)'"#$%&!'()$*++!X'(`J(Ka(b[cL@OX'(db()'
' PLR' I&+."1'T)'C$1".X'*6/.,&7'D2=&%X'/.1'^5/1'!/6&5")'U+.,".&/%'
6+6/,' -/%"/6"+.' 4/2&1' .+"2&' %&5+-/,' /,#+%"6=52)' ,-./!'
01)+%)*2#',-*)13!X'e@J(KaL[dcLebX'(ddL)'
' POR' ].6+.".' V=/54+,,&)' ].' /,#+%"6=5' >+%' 6+6/,' -/%"/6"+.'
5"."5"0/6"+.'/.1'/33,"8/6"+.2)'4!'52$-!'(326%)6'7%/!X'L@J(c
LKabdcd`X'L@@Q)'
' PQR' ].6+.".'V=/54+,,&'/.1'M+5/2'Z+8:)']'!"%26AD%1&%'Z%"5/,A
B$/,'],#+%"6=5' >+%' V+.-&S' Z%+4,&52' F"6=']33,"8/6"+.2' 6+'
T5/#".#)'4!'52$-!'(326%)6'7%/!X'Q@J(Ka(L@c(Q[X'L@(()'
' P[R' f+./6=/.'G)'N/%%+.'/.1'N&.'Z++,&)'M&'>/26'4",/6&%/,'2+,-&%)'
T.'8997X'L@(e)'
' PeR' ]./6' I&-".X' B/."' I"28=".2:"X' /.1' g/"%' <&"22)' V+,+%"0/6"+.'
h2".#'D36"5"0/6"+.)'T.':(;;<",=X'L@@Q)'
' P`R' f+./6=/.' G)' N/%%+.X' ].1%&F' ]1/52X' g"V=/.#' *="=X' /.1'
V/%,+2' _&%./.1&0)' !/26' 4",/6&%/,A23/8&' 26&%&+' >+%' 27.6=&6"8'
1&>+8$2)'T.'97,<X'L@([)'
' PbR' H$7' H",4+/' /.1' *6/.,&7' D2=&%)' U+.,+8/,' +3&%/6+%2' F"6='
/33,"8/6"+.2'6+'"5/#&'3%+8&22".#)'5>+$%/?2+*'51@*+!':%3>+!X'
`JOKa(@@[c(@LbX'L@@b)'
' PdR' ].6+."' N$/1&2X' N/%6+5&$' V+,,X' /.1' f&/.A\"8=&,' \+%&,)']'
.+.A,+8/,'/,#+%"6=5'>+%'"5/#&'1&.+"2".#)'T.'97,<X'L@@[)'
'P(@R' \/.$&,' <&%,4&%#&%X' M+5/2' Z+8:X' /.1' _+%26' N"28=+>)'
\+6"+.' &26"5/6"+.' F"6=' .+.A,+8/,' 6+6/,' -/%"/6"+.'
%&#$,/%"0/6"+.)'T.'97,<X'L@(@)'
'P((R' Z=","33' W%i=&.4j=,' /.1' k,/1,&.' W+,6$.)' ^l8"&.6' .+.,+8/,'
%&#$,/%"0/6"+.'>+%'+36"8/,'E+F)'T.'8997X'L@(L)'
'P(LR' ].6+.".'V=/54+,,&'/.1'Z"&%%&AI+$"2'I"+.2)'T5/#&'%&8+-&%7'
-"/'6+6/,'-/%"/6"+.'5"."5"0/6"+.'/.1'%&,/6&1'3%+4,&52)'0>3*#!'
52$-!X'`eJLKa(e`c(bbX'(dd`)'
'P(OR' B/-"1'I)'Z=",,"32)']'6&8=."9$&'>+%'6=&'.$5&%"8/,'2+,$6"+.'+>'
8&%6/".'".6&#%/,'&9$/6"+.2'+>'6=&'?%26':".1)'4'"95X'dJ(KabQc
d`X'(deL)'
'P(QR' ].1%&"' U":+,/&-"8=' G":=+.+-)' D.' 6=&' 2+,$6"+.' +>' ",,A3+2&1'
3%+4,&52' /.1' 6=&' 5&6=+1' +>' %&#$,/%"0/6"+.)' T.' A1B+2@.'
"B2@*3%%'02>BX'(deO)'
'P([R' *&/.' GF+5&7)' D.' 6=&' U$5&%"8/,' *+,$6"+.' +>' !%&1=+,5'
T.6&#%/,'^9$/6"+.2'+>'6=&'!"%26'W".1'47'6=&'T.-&%2"+.'+>'6=&'
I".&/%' *726&5'Z%+1$8&1' 47' m$/1%/6$%&)' 4'"95X'(@J(Kad`c
(@(X'(deO)'
'P(eR' *&/.'GF+5&7)'M&'/33,"8/6"+.' +>' .$5&%"8/,' ?,6&%".#' 6+' 6=&'
2+,$6"+.'+>'".6&#%/,'&9$/6"+.2'&.8+$.6&%&1'".'".1"%&86'2&.2".#'
5&/2$%&5&.62)'4!'C#2)B+!'()/$!X'L`dJLKad[c(@dX'(de[)'
'P(`R' N+447'C)'_$.6)'M&'/33,"8/6"+.'+>'8+.26%/".&1',&/26'29$/%&2'
&26"5/6"+.' 6+' "5/#&' %&26+%/6"+.' 47' 1"#"6/,' 8+53$6&%)' (888'
D#2)/!'913E>$!X'VcLLJdKab@[cb(LX'(d`O)'
'P(bR' ],&S/.1&%' f)' *5+,/' /.1' C"2"' W+.1+%)' W&%.&,2' /.1'
%&#$,/%"0/6"+.' +.' #%/3=2)' T.' F*2#)%)6' G*1#.' 2)@' H*#)*+'
52?-%)*/X'*3%".#&%X'L@@OX'3/#&2'(QQc([b)'
'P(dR' V/%,+'G+5/2"'/.1'C+4&%6+'\/.1$8=")' N",/6&%/,'?,6&%".#' >+%'
#%/7'/.1'8+,+%'"5/#&2)'T.'(997X'(ddb)'
'PL@R' k+ , : & % ' ] $ % " 8 = ' / . 1 ' f n % # ' <& $ , & ) ' U + . AI".&/%'H/$22"/.'!",6&%2'
Z&%>+%5".#'^1#&'Z%&2&%-".#'B"Y$2"+.)'T.'5>/$*#*#B*))>)6'
IJJKL'IM!'A";5N:.3E1/%>3X'(dd[)'
'PL(R' ].1%&F']1/52X'f+.#5".'N/&:X'/.1'\7&%2']4%/=/5'B/-"2)'
!/26' ="#=A1"5&.2"+./,' ?,6&%".#' $2".#' 6=&' 3&%5$6+=&1%/,'
,/66"8&)'913E>$!';#2E-!'C1#>3X'LdJLKa`[Oc`eLX'L@(@)'
'PLLR' !/.' C)' W)' V=$.#)' :E*?$#2+' ;#2E-' G*1#.)' ]5&%"8/.'
\/6=&5/6"8/,'*+8)X'(dd`)'
'PLOR' C"8=/%1'*".:=+%.)']'C&,/6"+.2="3'N&6F&&.']%4"6%/%7'Z+2"6"-&'
\/6%"8&2'/.1'B+$4,7'*6+8=/26"8'\/6%"8&2)'"))!'52$-!':$2$!X'
O[JLKab`ecb`dX'(deQ)'
'PLQR' Z&75/.'\",/.>/%)'*755&6%"0".#'*5++6=".#'!",6&%2)':("5'4!'
(326%)6':?%!X'eJ(KaLeOcLbQX'L@(O)'
'PL[R' ^1$/%1+' *)' I)' H/26/,' /.1' \/.$&,' \)' D,"-&"%/)' ]1/36"-&'
5/.">+,12' >+%' %&/,A6"5&' ="#=A1"5&.2"+./,' ?,6&%".#)' "95'
D#2)/';#2E-X'O(JQKaOOa(cOOa(OX'L@(L)'
'PLeR' ].1%&F']1/52X'U/6/2=/'H&,>/.1X'f&..">&%'B+,2+.X'/.1'\/%8'
I&-+7)'H/$22"/.'WBA6%&&2'>+%'>/26'="#=A1"5&.2"+./,'?,6&%".#)'
T.':(;;<",=X'L@@d)'
'PL`R' ].6+."+'V%"5"."2"X'G+47'*=/%3X'V/%26&.'C+6=&%X'/.1'Z/6%"8:'
Zo%&0)' H&+1&2"8' T5/#&' /.1' k"1&+' ^1"6".#)' "95' D#2)/'
;#2E-X'LdJ[Ka(OQa(c(OQa([X'L@(@)'
'PLbR' ^1$/%1+' *)' I)' H/26/,' /.1' \/.$&,' \)' D,"-&"%/)' B+5/".'
6%/.2>+%5' >+%' &1#&A/F/%&' "5/#&' /.1' -"1&+' 3%+8&22".#)' T.'
:(;;<",=X'L@(()'
'PLdR' _/.2' W.$622+.' /.1' V/%,A!%&1%":' <&26".)' U+%5/,"0&1' /.1'
1"Y&%&.6"/,'8+.-+,$6"+.)'T.'97,<X'(ddO)'
'PO@R' f/&2":' Z/%:X' _7&+.#F++' W"5X' g$ A<".#' G/"X' \"8=/&,' *)'
N%+F.X' /.1' T.' *+' WF&+.)' _"#=' 9$/,"67' 1&36=' 5/3'
$32/53,".#'>+%'OBAGD!'8/5&%/2)'T.'(997X'L@(()'
'PO(R' B/-"1' !&%26,X' V=%"26"/.' C&".4/8=&%X' C&.&' C/.>6,X' \/66="/2'
C$&6=&%X'/.1'_+%26'N"28=+>)'T5/#&'#$"1&1'1&36='$32/53,".#'
$2".#'/."2+6%+3"8'6+6/,'#&.&%/,"0&1'-/%"/6"+.)'T.'(997X'L@(O)'
'POLR' f"/;$.'I$'/.1'B/-"1'!+%276=)'*3/%2&'1&36='2$3&%'%&2+,$6"+.)'
T.'97,<X'L@([)'
'POOR' U",2'Z/3&.4&%#X'].1%o2'N%$=.X'M+5/2'N%+SX'*6&3=/.'B"1/2X'
/.1' f+/8="5' <&"8:&%6)' _"#=,7' /88$%/6&' +36"8' E+F'
8+53$6/6"+.' F"6=' 6=&+%&6"8/,,7' ;$26"?&1' F/%3".#)' ()$!' 4!'
913E>$!'7%/!X'e`JLKa(Q(c([bX'L@@e)'
'POQR' B/."&,'f)'N$6,&%X'f+./2'<$,YX'H/%%&66'N)'*6/.,&7X'/.1'\"8=/&,'
f)' N,/8:)']' ./6$%/,"26"8' +3&.' 2+$%8&' 5+-"&' >+%' +36"8/,' E+F'
&-/,$/6"+.)'T.'8997X'L@(L)'
'PO[R' f&%+5&'C&-/$1X'Z=","33&'<&".0/&3>&,X'p/"1'_/%8=/+$"X'/.1'
V+%1&,"/' *8=5"1)' ^3"8!,+Fa' ^1#&A3%&2&%-".#' ".6&%3+,/6"+.'
+>'8+%%&23+.1&.8&2'>+%'+36"8/,'E+F)'T.'97,<X'L@([)'
'POeR' m">&.#' V=&.' /.1' k,/1,&.' W+,6$.)' !$,,' E+Fa' D36"8/,' E+F'
&26"5/6"+.' 47' #,+4/,' +36"5"0/6"+.' +-&%' %&#$,/%' #%"12)' T.'
97,<X'L@(e)'
'PO`R' g".,".'_$X'C$"'*+.#X'/.1'g$.2+.#'I")'^l8"&.6'8+/%2&A6+A?.&'
3/68=5/68=' >+%' ,/%#&' 1"23,/8&5&.6' +36"8/,' E+F)' T.' 97,<X'
L@(e)'
'PObR' V=%"26"/.' N/",&%X' N&%6%/5' G/&60X' /.1' B"1"&%' *6%"8:&%)' !,+F'
?&,12a'B&.2&'8+%%&23+.1&.8&'?&,12'>+%'="#=,7'/88$%/6&',/%#&'
1"23,/8&5&.6'+36"8/,'E+F'&26"5/6"+.)'T.'(997X'L@([)'
'POdR' \+%"60' \&.0&X' V=%"26"/.' _&"3:&X' /.1' ].1%&/2' H&"#&%)'
B"28%&6&'+36"5"0/6"+.'>+%'+36"8/,'E+F)'T.';9,<X'L@([)'
'PQ@R' g$ ' I "X ' B +. # 4+ ' \ " .X ' \ " .= ' U ) ' B +X ' / . 1' f " /. # 4+ ' I $ )' ! / 26 '
#$"1&1'#,+4/,'".6&%3+,/6"+.'>+%'1&36='/.1'5+6"+.)'T.'8997X'
L@(e)'
'PQ(R' B&9".#' *$.X' *6&>/.' C+6=X' /.1' \"8=/&,' f)' N,/8:)' ]'
9$/.6"6/6"-&' /./,72"2' +>' 8$%%&.6' 3%/86"8&2' ".' +36"8/,' E+F'
&26"5/6"+.' /.1' 6=&' 3%".8"3,&2' 4&=".1' 6=&5)' ()$!' 4!' 913E>$!'
!
"`V@!
7%/!X'(@eJLKa(([c(O`X'L@(Q)'
'PQLR' Z=","33&'<&".0/&3>&,X'f&%+5&'C&-/$1X'p/"1'_/%8=/+$"X'/.1'
V+%1&,"/' *8=5"1)' B&&3!,+Fa' I/%#&' B"23,/8&5&.6' D36"8/,'
!,+F'F"6='B&&3'\/68=".#)'T.'(997X'L@(O)'
'PQOR' ]5"%' N&8:' /.1' \/%8' G&4+$,,&)']' >/26' "6&%/6"-&' 2=%".:/#&A
6=%&2=+,1".#'/,#+%"6=5'>+%',".&/%'".-&%2&'3%+4,&52)':("5'4!'
(326%)6':?%!X'LJ(Ka(bOcL@LX'L@@d)'
!
1
1. Experimental Results
In our main paper, we provided experimental results for a
number of vision-related inverse problems. 9is supplement
provides additional details on the formulations used, as well
as more extensive visual results for the experiments.
1.1. Disparity Super-resolution
For our disparity super-resolution experiment, we use the
dataset from [1], which is a subset of the Middlebury stereo
dataset. We show visualizations of our 16× super-resolution
disparity maps in Figure 4.
1.2. Optical Flow Estimation
In our experiments, we use the color-gradient constancy
model [2] instead of the brightness-constancy one [3]. In all
cases, one can express the optical flow data fidelity term as
() =
‖
(−
) +
‖
(S1)
see (27) in our main paper. 9e color-constancy model gives
us
=
,
=
(S2)
in which


denotes the - and the -derivatives of the
target image in the , and components, and

are
the difference of the reference image from the target one, in
the , and image components.
9e gradient-constancy model on the other hand gives us
the derivative data
=




,
=


(S3)
in which

,

and

are the second-order derivatives
of the target image, and

and

are the difference of the
first-order differences of the reference image from the target
ones. When the gradient constancy model is applied on each
of the color channels, we obtain
=












,
=






(S4)
in which we define the sub-matrices of and
similarly to
before.
Revaud et al. [4] use a weighted combination of two data
terms () based on (S2) and (S4). 9is combination can be
understood as forming new and
by stacking the ones in
(S2) and (S4). When the two data terms are combined using
equal weights, the inverse covariance matrix
∗
becomes
=
∑
∗
∗
∑
∗
∗
∑
∗
∗
∑
∗
∗
,
(S5)
and the transformed signal is
=
∑
∗
∗
∑
∗
∗
,
(S6)
cf. (27) in our main paper. In (S5)–(S6), the summations are
over the three color channels for each of the 0th, and the 1st
partial derivatives of the image. Figure 1 visualizes our flow
estimates.
1.3. Image Deblurring
Figure 2 provides crops of the deblurred images from the
the Kodak dataset [2], produced by different algorithms. We
optimize the algorithm parameters for the different methods
(Wiener, 2, and TV) via grid search. 9e Wiener filter uses
a uniform image power spectrum model. Note the use of the
bilateral filter is not optimal for de-noising as pointed out by
Buades et al. [5], who demonstrate the advantages of patch-
based filtering (nonlocal means denoising) over pixel-based
filtering (bilateral filter). Our deblurring results are based on
the bilateral filter, but one is free to use the non-local means
filter (or any other filter) for the de-noising operator .
Solving Vision Problems via Filtering
Supplementary Material
Sean I. Young
1
Aous T. Naman
2
Bernd Girod
1
David Taubman
2
sean0@stanford.edu aous@unsw.edu.au bgirod@stanford.edu d.taubman@unsw.edu.au
1
Stanford University
2
University of New South Wales
2
alley_1 (0010)
Blurred image
cave_4 (0038)
bamboo_
2 (0038
)
Geodesic
b
andage
_
1 (
0003
)
bandage_2 (0041)
Ground truth
Initial flow
Geodesic
Bilateral
Figure 1: Optical flow (top rows) and the corresponding flow error (bottom rows) produced using the geodesic and the bilateral variants of
our method. Whiter pixels correspond to smaller flow vectors.
3
Original
Blurred image
Wiener filter
𝐿2-regularized
TV
-regularized
Geodesic
Bilateral
kodim02
kodim04
kodim19
kodim24
Figure 2: Crops of images from the Kodak dataset when the B-spline blur kernel (𝑛= 8) is used. Our method exhibits less ringing compared
to the Wiener filter and the 𝐿2-regularization methods, and has less staircasing artifacts than the 𝐿1 (TV) method.
!
D!
['2!
16×
!
!
!
!
!
!
X%%O4!16×
!
!
!
!
!
!
@e6*&4!16×
!
!
!
!
!
!
!
N,5,',$?,!*()C,!
\'%&$0!2'&2:!0*4+)'*2<!
f%.7',4%3&2*%$!0*4+)'*2<!
Y&'!0*4+)'*2<!IC,%0,4*?K!
Y&'!0*4+)'*2<!I6*3)2,')3K
!
;*C&',!Db!9,!16×!4&+,'7',4%3&2*%$!0*4+)'*2<!()+4!+'%0&?,0!&4*$C!2:,!C,%0,4*?!)$0!2:,!6*3)2,')3!/)'*)$24!%5!%&'!(,2:%0!5%'!2:,!1088 ×1376!
4?,$,4!['2-!X%%O4-!)$0!@e6*&4!&4,0!*$!=">8!X,42!/*,.,0!%$3*$,!6<!B%%(*$C!*$8!!
2. Possible Limitations
! #$!J,?2*%$!D!%5!%&'!+)+,'-!.,!0*4?&44,0!2:)2!I"D)K!*4!/)3*0!
%$3<!.:,$!I"D)K!()2'*1!(𝐂+ 𝜆𝐋)
−1
!:)4!)!3%.7+)44!4+,?2')3!!
',4+%$4,8!A,!4:%.!2:*4!*$!;*C&',!D!I3,52K!5%'!2:,!?)4,!.:,',!
𝜆= 1!)$0!𝐂= 𝐈8!J*$?,!𝐂+ 𝜆𝐋!*4!J*$O:%'$7$%'()3*B,0-!*2!
:)4!)!:*C:7+)44!4+,?2')3!',4+%$4,!𝐼+ 𝜆𝐿
-!')$C*$C!5'%(!"!2%!
E8![4!)!?%$4,P&,$?,-!2:,!*$/,'4,!H32,'!',4+%$4,!(𝐼+ 𝜆𝐿)
−1
!
')$C,4!5'%(!1!0%.$!2%!`8Q8!A,!?)$!)++'%1*()2,!4&?:!)!H32,'!
',4+%$4,!)4!)!4&(!%5!3%.7+)44!)$0!)337+)44!',4+%$4,48!#$!%&'!
?%$2,12-!)$!)++'%1*()2*%$!%5!𝐮
opt
= (𝐂+ 𝜆𝐋)
−1
𝐂𝐳!?)$!6,!
%62)*$,0!&4*$C!)!?%$/,1!?%(6*$)2*%$!%5!𝐂𝐳!)$0!)!3%.7+)447
H32,',0!/,'4*%$!𝐀𝐂𝐳!%5!*28!Y$!2:,!%2:,'!:)$0-!*5!𝐼+ 𝜆𝐿
!*4!)!
3%.7+)44!',4+%$4,8!#$!2:*4!?)4,-!2:,!*$/,'4,!',4+%$4,!I4:%.$!
*$!;*C&',!D-!'*C:2K!*4!:*C:7+)44-!)$0!2:,!4%3&2*%$!𝐮
opt
!?)$$%2!
6,!)++'%1*()2,0!)4!)!?%$/,1!?%(6*$)2*%$!%5!𝐂𝐳!)$0!)!3%.7
+)447H32,',0!/,'4*%$!%5!*28!#$!+')?2*?,-!.,!?)$!42*33!&4,!I"D6K!
2%!4%3/,!2:,!2')$45%'(,0!+'%63,(8!!!
References
! =">! g),4*O! h)'O-! i<,%$C.%%! T*(-! Z&7A*$C! U)*-! @*?:),3! J8!
X'%.$-! )$0! #$! J%! T.,%$8! i*C:! P&)3*2<! 0,+2:! ()+!
&+4)(+3*$C!5%'!F]7UY;!?)(,')48!#$!ICCV-!E`""8!
! =E>! W*34! h)+,$6,'C-! [$0'j4! X'&:$-! 9%()4! X'%1-! J2,+:)$!
]*0)4-! )$0! g%)?:*(! A,*?O,'28! i*C:3<! )??&')2,! %+2*?! G%.!
?%(+&2)2*%$! .*2:! 2:,%',2*?)33<! k&42*H,0! .)'+*$C8! Int. J.
Comput. Vis.-!RLIEKb"D"S"Qa-!E``R8!
! =F>! X,'2:%30! T8! h8! i%'$! )$0! X'*)$! \8! J?:&$?O8! ],2,'(*$*$C!
%+2*?)3!G%.8!Artif. Intell.-!"LI"Kb"aQSE`F-!"ca"8!
! =D>! g,'%(,!N,/)&0-!h:*3*++,!A,*$B),+5,3-!l)*0!i)'?:)%&*-!)$0!
d%'0,3*)!J?:(*08!m+*?;3%.b!m0C,7+',4,'/*$C!*$2,'+%3)2*%$!
%5!?%'',4+%$0,$?,4!5%'!%+2*?)3!G%.8!#$!CVPR-!E`"Q8!
! =Q>! [$2%$*!X&)0,4-!X)'2%(,&!d%33-!)$0!g,)$7@*?:,3! @%',38![!
$%$73%?)3!)3C%'*2:(!5%'!*()C,!0,$%*4*$C8!#$!CVPR-!E``Q8!
!
;*C&',!F8!9,!5',P&,$?<!',4+%$4,!
(
𝐶+ 𝜆𝐿
)
−1
!?)$!6,!,1+',44,0!)4!
)!4&(!%5!3%.7+)44!',4+%$4,!𝐴
!)$0!)$!)337+)44!%$,!
𝐼
!%$3<!.:,$!2:,!!
',4+%$4,!
(
𝐶+ 𝜆𝐿
)
−1
!*4!3%.7+)4473*O,!I3,52K8!J:%.$!5%'!
𝜆= 1
8
𝐼+ 𝜆𝐿
!
(
𝐼 + 𝜆𝐿
)
−1
!
𝐴
!
𝐼
!
𝐼+ 𝜆𝐿
!
(
𝐼+ 𝜆𝐿
)
−1
!
𝐼
!